OpenAI Realtime API Review: Building a Production Phone Agent

I run an OpenAI Realtime voice agent in production for an energy company. It answers the phone, handles the first minute of the conversation, qualifies the caller, and hands off when it needs to. It sits behind a Twilio number. Total infrastructure cost is about $14 a month plus model usage. That number surprises people, so I want to explain what it actually buys you, and what it costs you in engineering time instead.

This is the platform I reach for when a client needs something a hosted voice-agent product cannot do, and when somebody (me) is going to own the code long-term. It is not the platform I reach for when a small business needs a receptionist live by Friday.

What the Realtime API actually is

The Realtime API is a speech-to-speech model you talk to over a persistent socket. You open a connection, stream audio in, and get audio back, with the model doing the listening, the thinking, and the speaking inside one system. There is no separate speech-to-text step feeding a chat model feeding a text-to-speech engine. That is the whole point of it.

That single detail is the reason latency feels different. In a classic pipeline you pay for STT, then the LLM, then TTS, and each stage adds its own wait before the next one can start. A speech-to-speech model collapses that. The agent starts responding while it is still processing, and interruptions land where a human would expect them to land. If you have ever sat through a voice bot that talks over you for two full seconds after you started speaking, you already understand why barge-in behavior matters more than voice quality for perceived intelligence.

The important framing: this is an API, not a product. There is no dashboard where a client edits the greeting. There is no built-in call log with transcripts formatted for a non-technical manager. There is no phone number. You bring all of that. What you get is the hard part (the conversation engine) and none of the easy parts.

The build: energy company, Twilio, one small server

The shape of the deployment is simple enough to describe in a paragraph. A Twilio number receives the call and opens a media stream to my server over a websocket. My server holds a second websocket open to the Realtime API. Audio frames go one way, model audio comes back the other way, and the server does format conversion in the middle because telephony audio and model audio do not speak the same encoding by default. Tools the model can call (lookups, sending a notification, ending or transferring the call) are handled server-side through function calling.

That is it. The server is small. It does not need a GPU. It does not need much memory. It needs to be close enough to the network to not add latency and stable enough to not drop sockets. Hence the $14.

This build replaced an older stack of mine that used Deepgram for transcription with separate model and voice stages. The migration was worth it for two reasons. First, the conversation felt more natural, particularly around interruptions and short acknowledgements. Second, there were simply fewer moving parts to fail. Every vendor in a pipeline is a vendor that can have a bad afternoon.

What I gave up in that migration was inspectability. In a pipeline stack I could see the exact transcript the LLM received, which made debugging "why did it say that" straightforward. With speech-to-speech, the model heard the audio. If it misheard a street name, there is no intermediate text artifact that proves it misheard. You get closer to the truth by listening to recordings than by reading logs, which is a genuinely different debugging posture and one I did not expect to matter as much as it does.

What it is good at

Latency and turn-taking. This is the headline and it delivers. Turn-taking feels closer to a phone call than to a kiosk. The agent handles "mm-hm" and mid-sentence corrections without falling apart.

Tone carried through the model. Because the model is generating audio rather than handing text to a separate voice engine, emphasis and pacing track the content. A pipeline TTS reads a sentence the same way whether the sentence is an apology or a confirmation. This one does not.

Control. Everything is yours. Session config, tool definitions, when to inject context, when to cut the model off, what to do with audio. If a client needs something odd (a specific compliance disclosure read verbatim before anything else, a hard cutoff after N seconds, a custom escalation rule) you can just write it.

Cost at volume. Usage is billed per unit of audio in and out, and that model rewards short calls with fast resolution. A one-minute qualification call is cheap. It rewards you further if you are disciplined about not stuffing the whole company handbook into the session on every turn.

What it is bad at, honestly

There is no product around it. No number provisioning, no call recording UI, no analytics, no non-technical editor, no knowledge base ingestion. If your client wants to change the greeting, that is a deploy unless you build a config layer. Build the config layer. I did, eventually, and I wish I had done it first. See chatbot analytics for what the reporting side usually needs to cover.

Telephony audio handling is fiddly. Sample rates, encoding, chunk sizing, and buffering are where the first week goes. None of it is intellectually hard and all of it is exactly the kind of thing that produces robotic clipping or one-second gaps if you get it slightly wrong.

Session length and reconnects. Long calls and dropped sockets need real handling. You need a reconnect path that does not lose conversational state, and you need to decide what the caller hears while you rebuild the connection. Silence is the worst possible answer, and silence is the default if you write no code.

No fallback comes for free. This is the same lesson I learned running voice agents on other stacks: when the primary model has a bad minute and you have not configured a fallback path, callers get dead air. Dead air reads as a broken phone line to a customer, not as an AI outage. If you take one operational thing from this page, take that one, and read voice agent dead air debugging before you go live rather than after.

Instruction adherence needs guardrails. Speech-to-speech models are conversational by nature, which is lovely until the agent conversationally invents a capability it does not have. I have had to fix a receptionist elsewhere that confidently told callers it had blocked a phone number. It could not block phone numbers. The fix was an explicit list in the prompt of abilities the agent does NOT have, stated as plainly as the abilities it does have. That technique transfers directly here. More on that in stopping chatbot hallucinations.

Cost, as of mid-2026

Two separate lines, and people conflate them constantly.

Line	Model	Rough shape
Model usage	Per unit of audio in and out, plus text tokens	Scales with talk time
Telephony	Per minute inbound, plus per number per month	Cents per minute
Infrastructure	Whatever you run the socket server on	Can be a very small box

My energy client's infrastructure line is about $14 a month. The telephony line for a set of small clients sharing one account runs around $130 a month across all of them. Model usage moves with call volume. The thing that blows up a budget is not the per-minute rate, it is a poorly designed agent that keeps callers on the line for four minutes doing something a two-turn exchange should have handled. Design for short calls. For the full picture on what clients actually pay, see how much does an AI chatbot cost.

When I choose it, and when I do not

I choose the Realtime API when the client needs behavior the hosted platforms will not give me, when call volume is high enough that per-minute platform markup matters, or when the agent has to sit inside an existing application rather than beside it. I also choose it when I am the one who will be maintaining it in two years, because API surfaces move slower than product UIs.

I do not choose it for a small business that needs a working AI receptionist quickly and cheaply, and where nobody is going to pay for ongoing engineering. For that, a hosted platform with a dashboard, built-in transcripts, and a knowledge base import wins on total cost of ownership every time, even if the per-minute rate is higher. Paying a platform a markup to not maintain socket reconnect logic is a good trade for most small deployments.

Testing it before it touches a real customer

The rule I hold everywhere applies double here: verify the wiring, not the logic. A local test harness that pipes clean audio into your socket proves your code parses frames. It does not prove Twilio's stream format matches what you assumed, it does not prove your tool call fires with real call metadata, and it does not prove the caller hears anything when the model stalls.

Call the number from your own cell phone. Then call it from a bad connection. Then interrupt the agent mid-sentence. Then say something outside its scope and see what it invents. I have shipped things that passed every automated check and still failed on the first real call because a variable that only exists on real calls was empty. Testing voice agents covers the full checklist I use.

FAQ

Is the OpenAI Realtime API cheaper than a hosted voice agent platform?

Per minute, usually yes, because you are not paying a platform markup on top of model and telephony costs. Total cost is a different question. Add the engineering hours to build socket handling, reconnects, logging, and an admin interface, then add ongoing maintenance. For a single small client, hosted almost always wins. For volume, or for anything custom, the API wins.

Can a non-technical person manage an agent built on it?

Not out of the box. There is no dashboard. If a client needs to edit prompts, swap greetings, or read transcripts, you have to build that layer yourself. Budget for it from day one, because the request always comes, usually in week three.

How does it compare to a Deepgram or STT-plus-TTS pipeline?

Speech-to-speech feels noticeably more natural, especially on interruptions and short acknowledgements, and there are fewer vendors in the path. The tradeoff is inspectability: a pipeline gives you an exact transcript of what the language model received, which makes some debugging much easier. I migrated a client off a pipeline stack and would do it again, but I keep call recordings because logs alone tell you less than they used to.

What breaks most often in production?

Connection handling and silence. Sockets drop, models occasionally stall, and if you have not decided what the caller hears in those moments, they hear nothing and hang up. Configure a fallback path, play something during reconnects, and monitor for calls that produce no agent audio at all.

Do I still need Twilio or Telnyx?

Yes, for phone calls. The Realtime API does not provide phone numbers or carrier connectivity. You need a telephony provider to receive the call and stream the audio, and you need to think about phone forwarding if the client already has an existing phone system in front of it.