ElevenLabs Agents vs OpenAI Realtime API: Which Voice Stack to Build On

These are the two voice stacks I use most, and they sit at opposite ends of the same trade-off. ElevenLabs Agents runs the receptionist on my own business line, a jewelry retailer's assistant across chat and voice on multiple sites, a UK windows-and-doors company's agent, and most of the demos I build for prospects. The OpenAI Realtime API, paired with Twilio, runs an energy company's voice agent that I migrated off an older Deepgram-based pipeline.

Both are in production right now. Both work. They are not interchangeable, and the reason is not audio quality, it is where the seams are.

The fundamental difference

ElevenLabs Agents is a product. You configure an agent: prompt, first message, voice, knowledge base, tools, turn-taking settings. Attach a number. Done. The vendor owns the pipeline between the caller's mouth and the model's output.

OpenAI Realtime is an API. It is speech-to-speech, which means audio in and audio out with the model reasoning natively over audio rather than a transcribe-think-speak relay. That is genuinely good technology. But it is a component, not a system. You still need telephony, a media bridge, a server to hold the session, tool execution, logging, and a deployment. You own all of it.

So the question is not "which sounds better." It is "how much of this stack do you want to own, and what do you get for owning it."

Comparison table

Dimension	ElevenLabs Agents	OpenAI Realtime API
What you get	A configurable hosted agent	A speech-to-speech model endpoint
What you build	Prompt, tools, knowledge base	All of that, plus the entire surrounding service
Time to first live call	Under an hour	A day or more the first time
Voice quality and variety	Excellent, large voice library, cloning available	Good, small fixed set of voices
Latency	Very good out of the box	Very good, but only if you manage every hop
Interruption and turn-taking	Tunable in the product	Model-native, but you handle the audio plumbing
Knowledge base	Built in, fast website ingest	You build retrieval yourself
Tool calling	Built in, triggers real actions	Built in at the API level, you execute and wire
Logging and transcripts	In-product	Whatever you log
Cost shape (as of mid-2026)	Per-minute agent usage	Per-minute audio tokens plus your own infra and telephony
Non-engineer editability	High	None, it is code
Failure mode	Confident hallucination, prompt-level bugs	Anything in a stack you built

Where ElevenLabs wins

Voice. It is the best-sounding stack I run, and on a real business line that changes caller behavior. People stay on longer and answer questions instead of hunting for zero. The voice library is deep enough that I can match a brand rather than settle. My process for picking one is in choosing a voice for your agent.

Speed. I can point a knowledge base at a prospect's website, have it crawled in minutes, write a prompt, attach a number, and hand over something callable before the sales conversation cools off. That has closed business for me. There is no version of the Realtime API that competes on this axis, because you cannot skip building the service.

Operational ergonomics. Transcripts, knowledge base updates, prompt edits, voice swaps: all in a console, all fast. When a client says "stop saying that," it is a two-minute fix, not a deploy.

Multi-surface. The jewelry retailer's assistant answers on chat widgets across several sites and on voice with the same brain. Rebuilding that consistency across a custom stack is real work.

Tools that do things. Sending an email, generating an image, booking a call: these run in production for clients from tool calls on ElevenLabs agents, and the wiring is not hard.

Where OpenAI Realtime wins

Control of the middle. With your own service between the caller and the model, you can inject context mid-call from your database, apply your own rules to what goes out, log every frame into your own analytics, and change any component without asking permission. For the energy company, that is exactly what let me migrate the speech stack underneath a running agent. The client noticed better calls and nothing else.

Cost predictability at your own scale. The infrastructure for that agent runs about fourteen dollars a month plus usage. That number surprises people who assume real-time voice needs heavy servers. It does not. The bill is dominated by per-minute audio and telephony, and owning the middle means you are not paying a product margin on top of the model.

Model-native audio reasoning. Because it is speech-to-speech rather than a chain of transcribe, generate, synthesize, the model hears tone and timing rather than a flattened transcript. In practice it handles overlapping speech and hesitation naturally, and it is not waiting on a transcription step before it can start thinking.

No product ceiling. Anything unusual, and voice projects get unusual, is possible because there is no vendor feature to wait for.

The costs of owning the stack

I want to be blunt about this because it is where people underestimate the project.

You own the latency budget. Every hop between the caller and the model adds milliseconds, and callers feel a delay long before they can name it. You own reconnects when a websocket drops mid-call. You own what happens during a provider outage, and if you have no fallback, the caller hears silence. Silence is the worst outcome in voice, because it looks like a broken phone line and nobody complains, they just hang up. I keep a full checklist in dead air debugging.

You also own testing, which is harder than it sounds. Simulators cannot inject system variables like caller ID, so an entire class of bug only appears on a real call. I call every agent from my own cell before rollout. Green tests are not evidence: I once shipped something with passing unit tests that failed on real production records and cost a client eleven leads. Verify the wiring, not the logic. That is the whole of testing voice agents.

The failure mode ElevenLabs has, and how to prevent it

Prompt-driven agents fail by inventing capability. The worst example I have shipped was a receptionist that told callers, with complete confidence, that it had blocked a phone number for them. It had no blocking tool. Nothing happened. The caller believed it.

The fix was not a better model or a stricter temperature. It was an explicit "abilities you do NOT have" list in the system prompt, naming the things people ask for that it cannot do and giving it the exact sentence to say instead. I now write that section into every prompt-driven agent, on either platform, before I write anything else. More in stopping chatbot hallucinations and the system prompt guide.

This failure mode exists on Realtime too. The difference is that with your own service in the middle, you have somewhere to enforce guardrails in code rather than only in words.

Pick ElevenLabs Agents if

• The phone is the product and voice quality is a competitive difference.
• You need something live this week, or you need a demo during a sales cycle.
• The job is answer questions, qualify, book, transfer. That covers most small-business work.
• Nobody on the project wants to be on call for a websocket.
• You want a non-engineer to be able to edit the prompt and knowledge base later.
• You are running the same assistant across chat and voice.

Pick OpenAI Realtime if

• You need to inject live data or business logic into the middle of a call.
• You want to own the pipeline so you can swap components later without a migration.
• You already run infrastructure and adding one small service is not a lift.
• Analytics, logging, or compliance requirements mean the transcript has to land in your systems, in your shape.
• Your call volume is high enough that shaving per-minute product margin matters.

Verdict

For client work, ElevenLabs Agents is my default, and it is not close. Most voice projects are a receptionist with tools, the voice quality is the part clients and callers actually notice, and getting to production in an hour instead of a week is worth more than the control I give up. The majority of my live agents on this side run there.

I reach for OpenAI Realtime when the middle of the call needs to be mine. The energy company's agent is the case: custom pipeline, custom data, a speech stack I have already swapped once, and infrastructure that costs about the price of lunch each month. If I had built that on a hosted product, the migration I did would have been a rebuild instead.

The trap is picking Realtime because it sounds more serious. Owning a stack is a maintenance commitment, and a voice agent that nobody maintains gets worse quietly. Pick the hosted product unless you can name the specific thing you need the middle of the call for. If you can name it, build it.

FAQ

Can I use ElevenLabs voices with the OpenAI Realtime API? Not in the native speech-to-speech mode, since Realtime generates audio itself from its own voice set. You can build a pipeline that transcribes, reasons with a text model, and synthesizes with ElevenLabs, but that gives up the latency and tonal advantages of speech-to-speech. Pick one architecture and commit rather than half-mixing them.

Which one handles a caller interrupting mid-sentence better? Both are good, for different reasons. Realtime handles it natively because the model reasons over audio and hears the overlap. ElevenLabs handles it because the vendor has tuned barge-in in a product you can adjust with settings. In a custom stack, interruption quality is only as good as your audio plumbing, so an unmaintained Realtime build will lose to a configured ElevenLabs agent.

What does a real deployment cost per month? The infrastructure is the small part. My Realtime plus Twilio agent runs about fourteen dollars a month in infrastructure plus usage, and voice usage across providers lands in cents per minute. Your bill is mostly minutes spoken, so the highest-leverage cost work is shortening greetings and stopping the agent from rambling. My cost guide breaks down how I quote these.

Do I need a phone provider with either one? Yes, always. Neither is a carrier. You need a number and telephony underneath, plus a plan for getting calls from the business's existing phone system to that number, which is frequently the fussiest part of the whole project. See phone forwarding for AI agents.

Can I start on ElevenLabs and move to Realtime later? The prompt, the knowledge content, and the list of tools port conceptually. Everything else is a rebuild, because you are going from configuring a product to building a system. Start hosted if you are unsure, but do not plan the migration as a small task. In practice, most agents that work never move.