Choosing a Voice for Your AI Agent Without Regretting It Later

Picking a voice feels like the fun part of building a phone agent. It is, until you realize the voice is also a latency decision, a comprehension decision, and a trust decision, and that swapping it after launch means re-testing everything. I have shipped agents on ElevenLabs, on Telnyx AI Assistants, and on an OpenAI Realtime stack over Twilio, and the voice choice has caused me more subtle production problems than the language model has.

Here is how I actually pick, in the order I make the decisions.

Start with the job, not the demo reel

The first question is what the agent does when someone calls, because that constrains everything downstream.

A receptionist that answers, qualifies, and transfers is having a short, functional conversation with someone who wants to get somewhere else. Warmth matters less than clarity and speed. This is most of what I build, and it is the pattern in the AI receptionist guide.

An outbound or sales-adjacent agent is asking for the caller's attention rather than responding to a request. The voice has to survive the first three seconds without triggering a hang-up, which mostly means sounding unhurried and unremarkable rather than impressive.

A shopping assistant that talks people through products, like the one I run for a jewelry retailer across chat and voice, needs a voice that can hold a longer turn without becoming tiring. This is where a lot of otherwise good voices fall apart, because a slightly nasal or slightly over-performed voice is fine for eight words and grating for eighty.

A support or clinical intake agent handles people who may be frustrated or unwell. Slower pacing, lower pitch variance, no perkiness.

Write down which of these you are building before you audition anything. It cuts the candidate list by more than half.

Latency is part of the voice

On a phone call, the perceived quality of an agent is dominated by how long it takes to start speaking, not by how pretty the timbre is. A gorgeous voice at 900ms time-to-first-audio feels worse than a plain voice at 250ms, every single time, because humans read silence on a phone line as confusion or a dropped call.

Three things drive this and they interact:

Model tier. Most TTS providers offer a quality tier and a low-latency or "turbo" tier. For real-time telephony I use the low-latency tier by default and only move up if a client can hear a problem. Nobody has ever heard the problem on an 8kHz phone codec.

Streaming. The voice has to stream audio out as tokens arrive rather than synthesizing a whole utterance first. If your platform does not stream, no voice choice will rescue you.

Geography. Synthesis region relative to your media server matters more than people expect. I have shaved a hundred milliseconds off a call just by moving the media path.

Also worth knowing: phone audio is narrowband. Whatever richness makes a voice sound premium in a browser demo is largely destroyed by the codec. Audition every candidate over an actual phone call, not through your laptop speakers. I keep a throwaway number pointed at a bare agent that reads a script, and I dial it from my cell for each candidate. That habit has changed my pick more than once.

Accent, register, and the local-business problem

The voice should sound like it could plausibly work at the business. That is the whole rule, and it is more constraining than it sounds.

For a UK windows-and-doors company, a bright American voice was a non-starter, and not because anyone objects to Americans. It signals outsourced call center, and it makes callers guarded before the agent has said anything useful. A neutral British voice with a slightly flat delivery did much better.

For a home-services contractor in a small US market, over-polished voices actively hurt. The voices that work sound like a competent office manager, not like a national brand's IVR. There is a real uncanny effect where an unusually smooth voice makes people suspicious that they are being sold something.

For a telecom client running multiple tenants, I ended up standardizing on a small set of two or three voices and assigning per tenant, rather than letting each new agent pick freely. That was a maintenance decision as much as a brand one. When you operate six or more live agents, a shared voice roster means a provider-side change is one investigation, not six.

Gender: I do not have a strong general finding, and I would distrust anyone who claims one from small samples. What I do is match the existing front-of-house of the business where there is one. If a real person currently answers that line, pick a voice in the same register.

The technical checks that actually catch problems

Once I have two or three finalists, I run them through a fixed gauntlet. These are the failure modes that show up in production and not in demos.

Numbers and identifiers. Have the voice read back a phone number, an order number, a dollar amount, a date, and an address with a unit number. Voices differ wildly here. Some read "1502" as "fifteen oh two" when you need "one five zero two." Some swallow the pause between digit groups so the caller cannot write it down. If your agent confirms appointment times or reads back a callback number, this test outranks everything aesthetic. The fix is often formatting in the prompt or in a tool response rather than a different voice, but you need to know which voice needs the least babysitting.

Proper nouns and brand names. Every business has at least one word the voice gets wrong. Test the business name, the product line, the street name, and any clinician or staff names the agent will say. Check whether your platform supports pronunciation overrides or a lexicon before you commit, because writing phonetic spellings into the system prompt is a workaround that gets messy fast. More on prompt-level fixes in the chatbot system prompt guide.

Barge-in. Interrupt the agent mid-sentence and see how it recovers. Some voice and VAD combinations produce a stutter, a double-start, or a stubborn refusal to yield. This is partly a platform behavior rather than a voice property, but voices with heavy breath and long trailing tails trigger more false endpointing.

Spelling out. Ask it to spell an email address. This is where a surprising number of otherwise excellent voices become unusable.

Long turn fatigue. Have it read ninety seconds of continuous copy. Listen to the whole thing over the phone. Any voice tic you notice at ninety seconds will drive callers up the wall at thirty.

Silence and filler. Check how the voice handles a filler phrase like "let me check that for you" while a tool call runs. If it sounds mechanical, callers assume a hold and start talking over it.

Voice cloning: usually not worth it

Clients ask for a cloned voice of the owner, or of the current receptionist, more often than you would think. I mostly talk them out of it.

The technical result is decent. The problems are elsewhere. You need documented, informed consent from the person whose voice it is, in writing, and you need to think about what happens when that person leaves the company. A cloned voice of a former employee answering the main line is a genuinely awkward situation, and it has a legal dimension I am not qualified to advise on. Cloned voices also tend to have narrower prosodic range than a well-engineered stock voice, so they can perform worse on exactly the number-reading and long-turn tests above.

Where I do think it earns its keep: a personal brand where the owner is the product, and the caller expects to hear that specific person. Even then, disclose that the caller is speaking with an automated assistant. Every agent I run discloses, without being asked, on the first turn.

Practical selection order

If you want the compressed version, this is my sequence.

Step	What I do	Why it is at this position
1	Define the job type	Eliminates most candidates instantly
2	Pick the low-latency model tier	Latency beats timbre on the phone
3	Shortlist 3 voices by accent and register	Fit to the business, not to my taste
4	Dial each one from a real cell phone	Codec destroys demo-quality differences
5	Run numbers, names, spelling, barge-in, long-turn	Catches the production failures
6	Have the client listen on their own phone	They hear their market better than I do
7	Lock it and document it	Voice ID goes in the config, not in someone's head

That last step matters more than it looks. Record the exact voice ID, the model tier, and the stability and similarity settings in the project config alongside everything else. I have had a provider deprecate or subtly re-tune a voice, and knowing exactly what was set before is the difference between a ten-minute fix and an afternoon of guessing.

Settings are part of the choice

Most providers expose knobs beyond the voice itself. Names vary, but the pattern is stability versus expressiveness, plus a similarity or clarity control and often a speed control.

Higher stability gives you consistent, slightly flatter delivery. For phone agents I bias toward stability, because an expressive voice that occasionally does something unexpected on a caller-facing line is a worse trade than a mildly boring one that never surprises you. Expressiveness is for content, not for service calls.

Speed is underrated. Nudging a voice slightly slower often reads as more competent and cuts down on callers asking for repeats, which shortens the call overall even though each turn is longer. Test it, then check the transcripts. Reading transcripts is how you find this stuff, and it is the core habit in chatbot analytics.

FAQ

Does a more expensive voice model actually sound better on a phone call?

Rarely enough that I default to the cheaper, faster tier. Telephony audio is narrowband and heavily compressed, which flattens most of the difference between quality tiers. Save the high-quality tier for agents that run in a browser or an app where the audio path is wide.

How much latency is too much before callers notice?

In my experience callers start reacting somewhere past half a second of dead time after they finish speaking, and by a second they will say "hello?" and talk over the agent. Aim to have first audio out fast, and use a short filler phrase before slow tool calls. Total silence with no explanation is the thing to eliminate, and if you have total silence on answer, that is a config bug rather than a voice problem, which I cover in voice agent dead air debugging.

Should the agent tell callers it is an AI?

Yes, and I build every agent to do it unprompted on the first turn. Beyond any disclosure rules that may apply where you operate, it sets expectations so callers speak more clearly and are less annoyed when the agent asks them to repeat something. It has never cost me a completed call.

Can I change the voice after launch without breaking anything?

You can, but treat it as a real change rather than a settings tweak. Re-run the number reading, name pronunciation, spelling and barge-in tests, because pronunciation workarounds written for the old voice may sound wrong in the new one. Then test-call the live number yourself before telling the client it is done.

Should each business get a unique voice?

Not necessarily. If you operate several agents, a small shared roster is much easier to maintain and lets you fix a provider-side issue once instead of per client. Reserve a distinctive voice for clients whose brand genuinely depends on it.