STT (Speech-to-Text)

STT (speech-to-text), also called ASR, turns the caller's audio into text the model can read. It is the first step in a voice agent's loop and the one most people ignore, right up until an agent starts confidently answering questions nobody asked.

Why it matters when you are buying or building

Everything downstream inherits the transcript. If STT hears "cancel" when the caller said "can I get," the LLM is not making a mistake when it responds about cancellations. It is answering the question it was given. When an agent seems to be going off the rails, read the raw transcript before you touch the prompt. Half the time the model behaved reasonably given bad input.

What degrades transcription in the real world:

• Phone audio. Narrowband, compressed, and sometimes routed through a carrier path that adds artifacts. Demos over a laptop mic are not representative.
• Background noise. Contractors call from job sites and trucks. That is your actual user.
• Accents and names. Proper nouns are the worst case, and they are exactly what you need for booking and lookups.
• Domain vocabulary. Product lines, clinical terms, street names. Most platforms let you supply hint words or a custom vocabulary. Use it.

What I have seen in production

Two practical lessons.

First, spelling matters more than you expect. Anything the agent has to capture exactly, email addresses, last names, order numbers, should be read back for confirmation. Do not trust a single-pass transcription of a name over a phone line. Building that confirmation step into the flow costs one extra turn and saves a wrong booking.

Second, the stack under you changes and it is worth revisiting. I run an OpenAI Realtime plus Twilio voice agent for an energy company that was migrated off an older Deepgram-based setup. The move simplified the architecture considerably, and the total infrastructure cost for that agent sits around $14 a month plus usage. Realtime-style models fold transcription and response into one pipeline rather than chaining separate STT, LLM, and TTS services, which removes hops and therefore latency. The tradeoff is less control over each individual stage.

Third, and this is the one that actually bites: you cannot validate STT in a simulator. Simulator testing does not carry real phone audio, and it cannot inject system variables like caller ID, so a class of bugs only ever appears on a real call. Every rollout I do includes calling the live number from my own cell, from a noisy place, saying the things a real customer would say. More in testing voice agents.

If the agent is misunderstanding people, the fix is usually upstream of the AI.