Latency

Latency is the time between the user finishing their turn and the agent starting to respond. In chat, a second or two is invisible. On a phone call, it is the single biggest factor in whether the caller believes they are talking to something competent.

Why it matters when you are buying or building

Human conversation runs on gaps of roughly a couple hundred milliseconds. Push past about a second of silence and callers start repeating themselves, which produces overlapping speech and makes everything worse. The failure compounds.

The delay is a sum, not one number:

Stage	What is happening
Endpointing	Deciding the caller actually stopped talking
STT	Finishing transcription
LLM	Time to first token of the reply
Tool calls	Any API lookup the agent makes mid-turn
TTS	Time to first audio out
Network	Round trips, including telephony

Two of these dominate in most builds. Tool calls are the worst offender, because a CRM or booking API that takes two seconds adds two seconds of pure dead silence. And endpointing is the one people never look at: an aggressive silence threshold makes the agent interrupt, a conservative one makes it feel slow, and it is usually a tunable setting. See turn-taking.

What I have seen in production

The moves that actually bought me time, in order of payoff:

1. Cut the prompt. Every token in a bloated system prompt is re-sent on every turn. Long prompts cost latency and money on each exchange, forever.
2. Cover slow tools with speech. If a lookup takes two seconds, have the agent say "let me check that for you" before it fires. Filled silence reads as normal. Empty silence reads as broken.
3. Stream everything. Streaming TTS starts speaking before the sentence is complete. Without it you pay for the whole generation before any audio moves.
4. Reduce hops. Migrating an energy client's agent onto an OpenAI Realtime plus Twilio stack collapsed several separate services into one pipeline, which is fewer network round trips by construction.

One warning: total silence is not a latency problem, and treating it as one wastes hours. If a caller is answered and hears nothing at all, that is a config or infrastructure fault, not slowness. I have chased what looked like a carrier issue that turned out to be an assistant configured with empty dynamic variables, and separately a missing fallback model during a provider outage. Both produce dead air that no amount of tuning will fix. That is its own debugging path: voice agent dead air debugging.

Measure latency on a real call, from a cell phone, on the client's actual number. Dashboard numbers do not include the telephony leg.