How to Build AI Agents for Healthcare (Step-by-Step)

Healthcare institutions process large volumes of clinical data, administrative tasks, client communication and operations. These tasks may often include some routine activities within EHRs, scheduling software, billing solutions, and other applications. Such agents can facilitate these processes as long as their actions stay within certain boundaries.

As opposed to a typical chatbot that is designed to answer questions, an AI agent can utilize approved apps, access information, follow the workflow instructions and perform actions towards achieving some particular goal. For instance, a patient follow-up agent can identify patients who need an appointment, check scheduling availability, send reminders and document everything in an authorized system.

Nevertheless, the healthcare field is a sensitive domain. The agent must protect personal information, distinguish reliable clinical data from generated data, and know when to stop and request human intervention.

In this manual, you will learn how to create an AI agent for your healthcare organization, from selecting an appropriate workflow to integrating and validating health data in the developed solution.

What Are AI Agents in Healthcare?

But before we start talking about how to create AI workflow agents, you need to learn more about healthcare AI agents. Healthcare chatbots or automation applications used for task completion can generally be described as AI agents. Yet, they are usually limited in usability.

Modern AI agents you plan to include in your practice are autonomous computer programs that use artificial intelligence to perform specific tasks and accomplish certain goals on behalf of a user, which is why they are called AI agents. Because these programs are intended to complete tasks associated with providing healthcare services, they will be referred to as healthcare AI agents.

For example, in March 2024, a mid-sized hospital network in Ohio rolled out an AI-based system that identified 340 instances of potential drug interactions in just the first 90 days of deployment. A pharmacist clinically reviewed and validated each flag before passing it along to the patient.

The development process took 14 months not because of the machine learning, but because of the engineering challenges of safely connecting to live clinical data via FHIR R4 and meeting HIPAA compliance requirements for access controls.

Where AI Agents Are Already Working in Healthcare

Healthcare agents can support several workflows. The appropriate level of autonomy depends on the potential consequences of an error and the reliability of the underlying systems.

Patient-facing: Scheduling, Intake, Reminders, and Triage Support

Patient-related agents could help arrange appointments, complete intake forms, provide instructions, and handle general reminders.

For instance, an appointment agent could access scheduling information, offer available appointment times, confirm the patient's chosen time, and document the booking process. An intake agent could gather information and note which fields require to be filled out prior to consulting a doctor. For the scheduling side, you can build an appointment scheduling agent that checks live availability before offering a slot.

Triage services will need additional security measures in place. An agent might gather information from a patient through an official survey and route the answers based on protocols set by clinicians.

Clinical Support: Documentation, Summaries, and Care-Plan Assistance

Clinical support personnel can summarize patient histories, arrange consultations, locate relevant records, and generate documentation drafts for clinicians to use.

A documentation agent may merge clinician input and validated EHR data in order to produce a draft of documentation. The clinician will review and correct any errors before signing off on the draft.

Care plan assistance can also help organize existing plans and identify gaps in information. It should be noted, however, that clinical decisions cannot be left to such automation.

Administrative: Prior Authorization, Claims, and Coding

Administrative processes may include repetitive documentation and data entry.

The agent that helps with prior authorizations may gather the necessary documents, check them against payer requirements, flag missing information, and generate a submission for staff to verify.

During claim processing, the agent may verify that the necessary information is entered and flag inconsistencies to pass to the billing department. Coding help can suggest the necessary codes for verification by qualified staff. For billing and claims follow-up, you can build a finance agent that flags inconsistencies for your billing team to review.

Operations: Referrals, Follow-Ups, and Population Health

Operational agents can coordinate tasks across departments and systems.

For example, a referral agent can verify that required documents are available, route a referral to the correct team, and flag cases that have remained unprocessed beyond a defined period. A follow-up agent can identify patients due for a scheduled check-in and prepare approved communications. If reminders are the first job you want to automate, an appointment reminder agent is a small place to start. Inferred path, so confirm it.

Population health workflows may use authorized data to identify records matching clinician-defined criteria. These results should be validated before they trigger patient outreach or changes to care.

How to Build an AI Agent for Healthcare: Step-by-Step

Building a healthcare agent involves more than connecting a language model to an EHR. The following steps provide a practical framework for developing a system that is useful, secure, and testable.

Source

Step 1: Define the Clinical Workflow and Set Its Safety Tier

The first step is to follow clinicians, map the workflow steps, identify where cognitive load is highest, and understand decision-making processes that are both frequent and rule-based.

At least one week should be spent observing the environment where the agent is going to be used. Note each decision point, data source, and interaction between systems. What is important to get is a process map that will answer three main questions:

Clinician Data: Chart review, laboratory results, images, medication history, and the ability to determine whether the patient is covered by insurance.

Medical decision-making: Dosage adjustment, referral, order entry, and documentation. Classify each decision into one of these categories:

• algorithmic (can be coded into rules)
• probabilistic (statistical analysis needed)
• judgmental (depends on experience and context)

AI agents should solve tasks only from the first two categories.

What happens when you make a wrong decision? Your safety level becomes either advisory, semi-autonomous, or fully autonomous with safeguards. If an AI clinical documentation agent gives the wrong suggestion, this is just inconvenient. If an AI recommends the wrong dosage for a blood thinner, it will kill a patient.

For example, in medication reconciliation, a typical workflow includes: the patient is admitted to the hospital, the nurse obtains a medication list from three sources (EHR, PBM, patient), the pharmacist checks for discrepancies, and the physician approves the list.

An AI agent solves the discrepancy check task, reducing the pharmacist's time from 22 minutes to 4 minutes per patient, according to data from ONC HealthIT pilot projects.

Step 2: Choose Your Build Approach and a HIPAA-Eligible Platform

Choose a development approach based on your technical resources, integration requirements, and expected level of customization.

Common options include:

• Custom development: Provides greater control over the agent’s architecture, workflow logic, integrations, and monitoring.
• Low-code or visual development: Helps teams assemble workflows and integrations with less manual coding, subject to the platform’s capabilities.
• Healthcare-focused platforms: May offer EHR connectivity, healthcare data models, or existing operational features.

Consider the options available for identity management, access controls, encryption, auditing and logging, data retention, incident response, and integration capabilities.

If you are a US healthcare provider managing electronic protected health information (ePHI) through cloud computing, make sure the provider can address any HIPAA requirements imposed on you by this activity.

A cloud computing vendor's marketing assertions about its cloud offerings are irrelevant to compliance. According to HHS, covered entities and business associates should address cloud computing risk and comply with applicable HIPAA requirements, including entering into a BAA if necessary. Read HHS guidance on HIPAA and cloud computing.

Also read: Our comparison of AI agent builders for platforms worth weighing before you choose.

Step 3: Set Up FHIR Data Access

Source

FHIR (Fast Healthcare Interoperability Resources) is an electronic healthcare data exchange protocol that describes healthcare information using structured data resources. All agents need access to clinical data to function in the healthcare domain. HL7 FHIR R4 specification is the de facto standard API for accessing and modifying clinical data. When working within Epic, Cerner (Oracle Health), or any other EHR certified to the ONC 21st Century Cures Act standard, the FHIR R4 endpoint is available.

In practice, FHIR data quality across different EHR implementations can vary dramatically. FHIR API provided by Epic produces well-structured US Core-compliant resources.

In some smaller EHRs, you might find technically correct FHIR with no codes, free text in place of coded data, and inconsistent terminologies. See the HL7 FHIR overview.

Depending on the workflow and the EHR’s implementation, useful resources may include:

• Patient: Demographic and patient-identification information.
• Appointment: Scheduling details and appointment status.
• Encounter: Information associated with a healthcare visit.
• Observation: Measurements, laboratory results, and other recorded observations.
• Condition: Documented health conditions.
• MedicationRequest: Prescribed medication orders.
• DocumentReference: References to relevant clinical documents.

Avoid assuming all EHRs provide the same resources and support similar activities. Check which FHIR version you have access to, which implementation guide it uses, which API endpoints are available, and any vendor restrictions.

Create an integration layer that lets you retrieve the necessary data, validate its format, and send back only the required fields. Make sure your implementation deals with possible failures, missing records, incomplete responses, and conflicts.

Step 4: Implement SMART on FHIR Authorization

FHIR supports data exchange, but applications still need an authorization mechanism to access protected information.

SMART on FHIR builds on OAuth-based authorization to manage application access and relevant launch context. Its scopes communicate the data and permissions an application requests. Review the SMART App Launch scope guidance.

Practice the least privilege concept, granting the agent only the rights that are necessary for its particular workflow process.

For instance, an appointment reminder agent might need to read appointment data and some restricted patient contact information, but not necessarily the ability to change medication orders or the clinical record.

Implementation considerations include:

• Ask only for the required scope of resources and operations.
• Follow the approved flows of authentication and secure token storage.
• Require proper context for users and patients where needed.
• Isolate the permissions for reading and writing resources.
• Limit access to sensitive operations and seek approval where required.
• Renew credentials following the security architecture design.
• Record access and relevant actions.

Avoid giving an agent broad access simply because it might be useful later. Additional permissions should require a documented need and a corresponding security review.

Step 5: Build the Agent’s Tools Using Verified Data

Healthcare agents must have structured tools that can call LLM functions. Tools must be designed as functions with properly defined input types, output validation, and proper error handling.

That’s when frameworks such as LangChain or LlamaIndex come in handy, though in most cases healthcare teams build their own tooling layer to maintain better control over PHI management.

The design philosophy behind healthcare tools is very strict: each tool should output structured information from a trusted source without ever allowing the LLM to produce information based on its parametric knowledge.

The central rule is that generated text must never substitute for authoritative healthcare data.

Step 6: Add Clinical Safety Guardrails

This is the point where healthcare AI agents differ fundamentally from general AI systems. All agent outputs should pass clinically approved safety checks before being presented to a clinician or used downstream.

This guardrails layer does not constitute an additional step in the process; it is a critical part of the architecture itself.

Important safeguards include:

• Scope restrictions: Prevent the agent from taking actions outside its approved workflow.
• Input validation: Check that information is complete, relevant, and correctly formatted.
• Source grounding: Require clinical facts to come from authorized, verifiable sources.
• Uncertainty handling: Escalate ambiguous requests, conflicting records, or missing information.
• Urgent-case routing: Follow approved escalation protocols when a patient reports potentially urgent symptoms.
• Action limits: Restrict repeated tool calls, transaction size, and the number of steps.
• Privacy protections: Prevent unauthorized disclosure of patient information.
• Prompt-injection defenses: Treat retrieved documents and external content as data, not as instructions that can override system controls.

Where clinical decisions are involved, have qualified clinicians define and review the relevant protocols. Assess whether the intended function may trigger additional regulatory requirements before deployment.

Step 7: Add Human-in-the-Loop Review

The human review process is especially important when the agent produces clinical documentation, evaluates ambiguous data, or takes an action with implications.

Decide on the correct review point for each workflow. For example, an administrative agent may automatically send out a previously reviewed appointment reminder once the data have been validated. A clinical documentation agent will usually require the clinician's approval before finalizing its work.

A useful approval workflow should display:

1. The action the agent proposes.
2. The verified information supporting the action.
3. Any missing fields, uncertainties, or conflicting sources.
4. The potential consequences of proceeding.
5. The available approve, edit, reject, or escalate options.

Record the reviewer, decision, timestamp, and resulting action where appropriate. Make it easy for staff to correct errors and report unsafe behavior.

Human oversight must be meaningful. A reviewer needs sufficient information, authority, and time to challenge the agent’s output rather than simply approving every recommendation.

Step 8: Deploy With Clinical and Infrastructure Monitoring

Once tested and integrated, deploy the agent in a controlled environment. Start from simulation or de-identified data where applicable, then move on to a pilot with production access and governance approval.

Monitor both technical and workflow performance.

Technical measures could be API failures, latency, tool-call errors, authorization failures, and service availability. Workflow measures could be task completion, referral turnaround, document correction rate, and number of escalations.

Safety monitoring could focus on improper data acquisition, unsupported statements, inappropriate actions, missed escalations, and any other incidents relevant to the use case.

Keep audit logs while adhering to relevant privacy and access controls. Develop an incident response protocol that lets the team shut down the agent, stop access, investigate the problem, and safely bring the service back oback online

The monitoring process should also account for changes in the EHR interface, clinical guidelines, model performance, and data quality. A system that was working well during testing might fail once its environment changes.

Step 9: Run Clinical Validation Before Going Live

Technical testing alone cannot establish that a healthcare agent is safe or useful. Validate the system against realistic workflows and clinically relevant failure cases.

Create a test set that covers normal requests, missing information, ambiguous instructions, conflicting records, unusual scenarios, and attempts to exceed the agent’s permissions.

Phase 1: Testing with Synthetic Data. Test the agent on anonymized patient datasets. Synthea can generate realistic FHIR bundles for testing. Generate at least 10,000 synthetic patients representing the diversity your agent would face in production.

Test accuracy, precision, recall, and F1 score using the known-correct clinical decisions. Aim for an F1 of at least 0.92 in the advisory tier and 0.97 in the semi-autonomous tier.

Phase 2: Shadow Mode. Deploy the agent side by side with clinicians, but keep its output hidden from them. Document what the agent would have done in relation to the real decisions taken by the clinicians. Run the shadow mode for at least 30 days or 1,000 patient encounters, whichever comes first.

Evaluate discrepancies between the agent’s and the clinicians’ decisions and classify each one as an agent’s right call, a clinician’s right call, and an ambiguous case. Here, identify any FHIR-specific data quality issues that the synthetic data does not show.

Phase 3: Supervised Pilot. Show the agent's suggestions to trained doctors, who must then evaluate all of them. Write feedback systematically for all of the suggestions made by the agent.

Phase 4: Deployment under Supervision. Deployed with continuous statistical monitoring. Make use of CUSUM analysis to detect any drop in performance. Establish a quarterly clinical panel to assess the agent's performance through quality measures.

Build vs Buy for Healthcare Agents

Before developing your own custom healthcare AI agent, consider buying one, since many AI agents already exist in the market, such as healthcare chatbots.

But if you have different requirements, building a custom healthcare AI agent is the only option. But regardless of that, to make your decision about building or buying easier, take a look at the table below:

A hybrid strategy could also be implemented. The company could use an established platform for identity management and connectivity of the EHR system, but develop its own solutions and logic for a particular application. If you want to build the custom part yourself, an AI agent builder lets you describe the workflow in plain language and set the approval rules.

Assess the vendors based on their capability, integration compatibility, data handling capabilities, access control, audit capabilities, availability, and commitment. It should not be assumed that procurement of the healthcare product will transfer all compliance obligations to the vendor.

How to Develop a Healthcare AI Agent With Emergent

Emergent’s healthcare agent builder can help teams prototype AI healthcare applications by designing workflows and refining the application with prompts. It may be a good option to create administrative agents for appointment scheduling, intake processes, and referrals.

To do that, follow the steps below:

Describe the use case: Determine which workflow is needed, find out who will use it, and set the agent's rights and escalation rules.

Develop the application: Build the workflow in Emergent, including user roles, validation rules, and approvals.

Connect healthcare systems: Connect the supported APIs and services based on FHIR where possible. Implement security features and least privilege access.

Implement the safety measures: Provide data validation, action validation, logging and human approval for significant actions if required.

Test and roll out: Test with synthetic or de-identified data, run error analysis and permission tests, and measure performance in a limited rollout.

Emergent may provide application development, yet HIPAA compliance, secure EHR integration, and clinical validation should not be taken for granted. Confirm platform capabilities and service agreements before processing PHI.

Discover Emergent to find out more about its AI application development features.

Conclusion

Developing AI agents for healthcare needs a fine balance between automation, interoperability, data security, and clinical governance. The best approach is to start with a clear workflow, connect the agent to credible data sources, restrict agent permissions and set clear rules for review and escalation.

FHIR and SMART on FHIR can assist in managing data transfer and restricted access, and monitoring and clinical validation can be used to determine if the system operates as designed. HIPAA requirements and vendor contracts must be addressed when using protected health information.

Rather than starting with a fully automated clinical system, facilities can begin with a limited workflow, monitor performance, and gradually scale up based on evidence of success. It will enable healthcare facilities to automate healthcare reliably without making patient safety an afterthought.