Voice AI for Healthcare — Molly Morrell

PolyAI builds voice AI agents that answer customer service calls for large companies, from banks to hotel groups to healthcare providers. As an Agent Designer, I designed and built those agents end to end: the conversation design and call flows, the Python functions and API integrations behind them, testing, and launch with the client. I ran several client accounts at once.

This case study is one of them: a voice agent that answers a US healthcare provider’s appointment line. It identifies the caller, works out what kind of appointment they need, checks insurance and availability in the clinic’s records system, and books it. I took it from build through testing to go-live in June 2026 with one engineer, then led the fixes through hypercare.

My part: I designed the call flows and error states, wrote the conversation design, built much of it in Python, and used live call data to find where callers were getting stuck.

The challenge

Booking a healthcare appointment by phone is full of edge cases. The agent has to confirm who’s calling before it can touch their record. It has to tell a sick visit from a recheck or a hospital follow-up, check the patient’s insurance is accepted, and find a slot with the right provider. Every one of those steps can fail.

And the people calling a clinic are exactly the callers voice AI finds hardest. Many are older, quietly spoken or hard of hearing, and the technology works against them in specific ways:

Quiet speech goes unheard

Voice activity detection decides when someone is talking. Soft voices can fall below its threshold, so the agent hears silence, re-prompts, or talks over the caller.

Pauses get cut off

Callers who pause mid-answer, such as halfway through a date of birth, can hit the end-of-speech timeout and get cut off before they finish.

Phone audio loses detail

Phone lines carry narrowband audio that strips out the high frequencies in sounds like s, f and th. Letters like B, D and P blur together, which is hard on names and spelling.

Shared phones, unusual names

Family landlines have several patients on one number, and surnames are often words speech recognition has never seen. Both make identity checks fail.

Designing the booking flow

I mapped the full appointment journey in Lucid. Every step pairs a conversation with a call to the clinic’s records system, and every step has its own ways to fail.

Six steps, and what happens when each one fails

Pick a step to see the conversation, the system call behind it and its failure paths. Function names are simplified.

01Identify the caller

The agent

Looks the caller up from their phone number, name and date of birth
Confirms who's calling before touching any record

System call

Patient search in the clinic's records system

If it fails

Several patients on one line → narrow it down with date of birth
Name misheard → spell it back. I proposed accepting a near-match once date of birth has already found one patient, which is now product roadmap work
No match → treat as a new patient, or hand over

Designed for

Speech recognition tuned for names and dates of birth

02Appointment type

The agent

Works out what kind of appointment the caller needs, in plain language

System call

Appointment rules for each visit type and provider group

If it fails

Caller isn't sure → offer simple options instead of medical terms
Needs a clinical decision → hand over early, with the reason attached

Designed for

Short questions, one decision at a time

03Insurance

The agent

Checks the plan on file before looking for slots, so nobody books an appointment they can't use

System call

Insurance on the patient record, matched to accepted plans

If it fails

Plan not accepted, or provider not credentialed → explain why, and pass to the scheduling team with a reason code

04Find a slot

The agent

Offers a couple of times at once rather than reading out a long list

System call

Availability search by visit type, provider group and date window

If it fails

No slots in the window → widen the search to other providers in the same group
Caller turns down several times → after a set limit, hand over instead of looping

Designed for

Two options at a time is easier to hold in your head

05Book

The agent

Books the chosen slot while the caller waits

System call

Create the appointment in the records system

If it fails

Slot taken mid-call → try the next candidate automatically
System times out → check whether the booking went through before retrying, so nobody is double-booked

06Confirm

The agent

Reads back the day, time and provider, then offers to repeat

System call

Booking details from the created appointment

If it fails

Caller didn't catch it → repeat slowly, in smaller pieces

Designed for

Read-backs for anyone hard of hearing

Alongside booking, I designed and built the related flows:

Self-serve cancellationAppointment confirmationPrescription refills by SMS linkCallbacks for returning callersPaediatrics, including siblings back to back

Under the hood

The agent sits on PolyAI’s platform, but the logic that makes it work for this clinic is code I wrote and integrations I configured.

From phone line to metrics

The layers a call passes through. The middle three are where I spent most of my time.

Telephony
Calls arrive over SIP from the clinic's phone tree. Handovers are SIP transfers to the right staff queue, with a reason code attached.
Speech in
Speech recognition tuned for phone audio, with recognition biased per step: names, spelled letters, dates, yes/no. Voice activity and silence timeouts tuned per step too.
Reasoning
A customer-service LLM follows the flow, decides what to say, and calls tools. Text-to-speech speaks the reply.
Flows and functions
The flows I designed, built as Python functions: identity checks, routing, slot search, booking, silence handling and reason codes. Version-controlled, reviewed in pull requests.
Integrations
REST calls to the clinic's EHR / practice management system for patients, insurance, availability and appointments, plus SMS links for refills.
Quality and metrics
Automated QA on test calls, live call review, and the success and handoff metrics I defined.

One booking, end to end

The API calls behind a single booking, including a timeout that could have created a duplicate.

GETPatient searchby phone number, then name and date of birth
GETPatient insurancesplans on the record, matched to accepted plans
GETAvailabilityby visit type, provider group, location and date window
POSTCreate appointmentthe chosen slot, with visit type and reason
PUTCancel or reschedulean existing appointment, with a cancel reason
GETPatient appointmentsused to check for a booking before retrying

Caller

Voice agent (ASR · LLM · TTS)

My Python functions

EHR / practice management API

"I need to book a sick visit"

start_booking_flow()

GET patient search: phone, then name + date of birth

one patient record

GET insurances on the patient record

plan accepted

GET availability: visit type, provider group, date window

three open slots

offers two times

"Tuesday at ten"

POST create appointment

no response: timed out

GET patient appointments: is that slot already booked?

yes: booking exists

reads back day, time and provider

Designing for when things go wrong

Most of the design work is in the error states. Each way a booking can fail gets its own path and its own reason code, so the agent recovers where it can, hands over cleanly where it can’t, and we can measure which is which.

No slots in the window

Widen the search to other providers in the same group before giving up.

Slot taken mid-call

Try the next candidate slots automatically instead of failing.

Records system times out

Check whether the booking went through before saying it failed, so nobody is double-booked.

Needs a person

Insurance, chart alerts or timing rules: hand over early, with the reason attached.

The timeout check came from my own investigation. I traced 33 duplicate bookings over about 30 days to the scheduling integration: reschedules weren’t updating the appointment record correctly, so the system accepted a second booking for the same patient. Checking for an existing booking before trying again closed the gap on our side while the integration fix was worked through.

Fixing speech recognition problems

Live calls showed where ASR was letting callers down, and I made targeted fixes:

Names and dates of birth. I tuned speech recognition for these specifically, including a new function that biases ASR differently at each step of the call.
Silence. Silence handling was originally a set of prompt rules. I replaced it with a stateful function, which also cut the agent’s rules and prompt by 47% (695 lines down to about 369).
Name matching. I found that name capture caused about half of failed identity checks. Lenient matching only applied when a phone number had a single patient on it, so family and shared lines were held to a strict bar even after phone number and date of birth had already identified the patient. I proposed applying lenient matching whenever date of birth gave a unique match, and that became product roadmap work.

Listening for the right thing

Speech recognition is biased towards what each step expects, and silence is handled as a state, not a rule.

Name step

Biased towards names and spelled-out letters

Date of birth step

Biased towards months, days and years

Confirmation step

Biased towards yes, no and corrections

Listening

Making it work for more callers

To make the agent usable for older, quietly spoken and hard-of-hearing callers, I made four changes:

What it sounds like

Placeholder dialogues until the real test-call transcripts and audio are added.

Agent

Can I take your date of birth, please?

Six seconds of silence

Agent

No rush. Just the month, day and year you were born.

Shorter and rephrased, not the same prompt again

Caller

March the… third. Nineteen fifty-one.

Agent

Thank you. That's March third, nineteen fifty-one?

Key details read back

Caller

Yes, that's right.

Agent

And what's the patient's last name?

Caller

Nowak. N, O, W, A, K.

Speech recognition hears "Novak". Date of birth has already matched one patient on this number

Agent

Thanks. Just to check, is that Anna Nowak, born June twelfth?

A near-match is confirmed with the caller, not rejected

Caller

Yes.

Caller

Tuesday at ten works.

Agent

Great, I'm booking that now.

The records system doesn't respond in time

Agent

Bear with me one moment while I check that's gone through.

Checks for the booking before trying again

Booking found

Agent

You're all set for Tuesday at 10am at your usual clinic.

No duplicate, and no false "something went wrong"

Measuring what matters

Two metric fixes made the numbers trustworthy:

I fixed the booking-success metric so it only counted calls that had passed identity checks, which had been distorting the numbers.
I built a new “preventable handoff” metric that separates transfers the agent should have handled from the ones it shouldn’t.

With the metric fixed, the improvements showed up: over August booking success rose from about 50% to 58%, and cancellation success climbed from 48% to over 60%, helped by self-serve cancellation for every appointment type, which I built.

56–58%

booking success in late August and early September 2026

60–65%

the client's target for booking success

~50%

of failed identity checks traced to name capture

Booking success was close to target but not there yet. The gap came from three places, and each one shaped what I proposed next:

Identity checks

Name capture caused about half of failed checks, and family lines were held to the strictest bar.

Out-of-hours calls

Callers completed identity checks and booking steps, only to hit a dead end with no one to hand over to.

Unsupported visit types

Some appointment types weren't supported by the booking integration, so those callers had to be transferred.

Preventable handoffs

The new metric showed which transfers the agent should have handled, giving a clear list of fixes.

What’s next

From reviewing calls in September, I proposed:

Shorter identity checks. Look the caller up by phone number up front, which skips the “new or existing patient?” question. Skip the date-of-birth read-back when only one patient matches. Accept a near-match on the name once date of birth has found one patient.
Out-of-hours calls. Set expectations in the greeting, and send incomplete bookings to the scheduling team for a next-day callback instead of a dead end.
Fewer preventable handoffs. Work through the preventable-handoff list, starting with the most frequent reasons.

What I learned

Voice UX

Design the failure paths first. The happy path is the easy part, and most calls leave it at least once.
Every prompt is a cost. Shorter, simpler questions helped every caller, not just the ones who struggled.

Accessibility

Voice design is accessibility design. The callers ASR struggles with are the ones who most need the phone to work.
Silence isn't nothing. A caller who goes quiet is often still thinking, and the agent has to give them room.

Building

Some problems that look like conversation design are integration problems. The duplicate bookings were about how the agent worked with the scheduling system, not what it said.
Moving logic out of the prompt and into code made the agent more reliable and easier to change.
Fix the metric before you chase it. A miscounted success rate sends you after the wrong problems.