Your Voice Agent Has 800 Milliseconds. Everything Else Is a Negotiation With Physics

Why Did Voice Suddenly Become a Production Problem?

For two decades, automated phone systems were a solved and universally disliked category. You pressed 1 for billing, you said the word representative four times with increasing hostility, and eventually a human picked up. The technology was rigid, the intent models were shallow, and everybody involved understood the arrangement. Then speech-capable language models arrived, the demos became genuinely convincing, and every board in Europe asked the same question in the same quarter: why are we still paying people to answer the phone?

The forecast that shaped most of those conversations came from Gartner's prediction that conversational AI would reduce contact centre agent labour costs by $80 billion in 2026, with conversational AI handling one in every ten agent interactions by this year, up from an estimated 1.6% at the time of the forecast. That is a twenty-fold increase in automation share, and it was published before the current generation of speech models existed. Nobody in that room was being unreasonable.

What has actually happened is more interesting than either the hype or the backlash. The State of Voice AI survey of 400 business leaders, conducted with Opus Research, found 67% of organisations treating voice AI as core to their product and business strategy and 84% intending to expand voice AI spending over the following year. The same survey found that 80% already operate traditional voice agent systems, and that only 21% are very satisfied with them. Strategy is settled. Satisfaction is not.

That gap is where the engineering lives, and it is unusually easy to misdiagnose. When a voice deployment disappoints, the instinct is to blame the model, switch vendors, and rewrite the prompt. In our experience the model is rarely the binding constraint. The binding constraints are latency, interruption handling, speech recognition on real-world audio, and the absence of any test suite that can tell you whether last week's change made the agent worse. Those are not AI problems. They are the problems of building a soft real-time distributed system, and they have been well understood in telephony engineering for thirty years.

What Makes a Voice Agent Different From a Chatbot?

A text chatbot that takes four seconds to answer is a slightly slow chatbot. A voice agent that takes four seconds to answer is a broken phone call. That single difference cascades into every architectural decision, and it is the thing most teams discover only after they have built the wrong shape of system.

Human conversational turn-taking runs on gaps of roughly 200 milliseconds. Push much past a second and the caller assumes the line has dropped, starts talking again, or says hello into the silence, which then arrives as new input and derails the agent's next turn. The practitioner benchmarks in this guide to voice agent evaluation metrics place a 95th-percentile response under 800 milliseconds in the healthy band, treat 800 to 1500 milliseconds as a warning, and flag anything above 1500 milliseconds as a failure condition, while putting the industry median somewhere between 1.4 and 1.7 seconds. Read that carefully: the typical deployed voice agent sits in the warning band on a good day.

The arithmetic explains why. A cascaded pipeline has to detect that the caller stopped speaking, finish transcribing, send the transcript to a model, wait for enough tokens to start synthesising, generate audio, and push that audio back through a codec and a carrier network that adds its own jitter. Each stage is fast in isolation and each stage has a tail. Chain them sequentially, wait for each to complete before starting the next, and you will land at three to four seconds without anyone writing slow code. Every serious treatment of latency budgets for real-time voice arrives at the same conclusion: you cannot optimise your way to sub-second from a batch architecture. You have to stream at every boundary.

Then there are tool calls, which is where most production budgets actually die. The agent has to look up an order in an ERP that answers in 900 milliseconds on a good day, or authenticate a caller against a system that was designed for a web form. That call alone consumes the entire budget. The techniques that work are the ones telephony engineers would recognise: dispatch the lookup speculatively while the caller is still finishing their sentence, generate acknowledging speech to cover the wait, and treat a slow backend as a first-class product decision rather than a latency bug to be fixed later. What you cannot do is put a synchronous enterprise API call in the middle of a conversational turn and hope.

Why Does Turn-Taking Break Every Demo?

Watch a voice agent demo and you will notice something: the presenter waits politely for the agent to finish. Real callers do not. They interrupt, they say yes halfway through the confirmation, they talk over the agent to correct a misheard postcode, and they make small acknowledging noises that mean keep going rather than stop. Handling this is called barge-in, and it is the single hardest product problem in the category.

The mechanical part is tractable. When the caller starts speaking, stop the synthesised audio immediately, flush whatever is already buffered in the playback pipeline, and cancel the in-flight model generation so you are not paying for tokens nobody will hear. Teams get this working in a week. The part that takes months is the state reconciliation underneath it, and it produces the strangest bug reports in the whole discipline.

Here is the failure. Your agent generated a sentence containing the customer's account balance. Two hundred milliseconds of it reached the caller's ear before the interruption cut it off. Your conversation history, however, records the entire sentence as spoken. The agent now believes the caller knows their balance. Three turns later it says something like, as I mentioned, that leaves you short by forty euros, and the caller has no idea what it is talking about. Nothing crashed, no exception was logged, and your transcript looks perfectly coherent. Losing the distinction between generated and actually heard is an architecture defect, not a model limitation, and it is invisible to every text-based test you have.

The detection side is a genuine tradeoff with no free answer. Threshold-based voice activity detection is cheap and fast but cannot tell a thoughtful pause from a finished sentence, so it either interrupts people mid-thought or leaves dead air after every utterance. Semantic turn detection models predict whether a speaker has finished based on what they said rather than how long they have been silent. Comparisons of the major open-source stacks in this review of Pipecat and LiveKit as voice orchestration frameworks put the difference at roughly a 30% reduction in the agent talking over the user, with better behaviour on short utterances. It is a real improvement and it is still a probabilistic classifier making a judgement call several times a second on a live phone call.

This is also the reason the build-versus-buy decision is not obvious. Turn-taking, interruption handling, telephony transport and function calling now ship as primitives in the mature frameworks. That is a genuine head start, and it is not the same as a solution, because the tuning of those primitives against your callers, your audio conditions and your tolerance for interruption is where the remaining months of work sit.

Whose Voice Does Your Speech Model Actually Hear?

Speech recognition benchmarks are quoted on clean, read, native-accented audio in a quiet room. Your callers are in cars, in warehouses, on speakerphone, in a second language, spelling a Serbian surname to a model trained mostly on English, and doing it over a narrowband telephony codec that discards a large part of the frequency range before your system sees a single sample.

The disparity is measurable and it is not new. The Stanford-led study published in PNAS on racial disparities in automated speech recognition tested five leading commercial systems and found an average word error rate of 0.35 for black speakers against 0.19 for white speakers, on identically structured interviews. Nearly double the error rate, from the same vendors, on the same task. Anyone deploying a voice agent to a general population should assume their headline accuracy number describes their best-served callers and nobody else.

For a European deployment the problem compounds in a specific way. Your callers code-switch mid-sentence, they pronounce product names in three different accents, and a meaningful share of your calls consist of alphanumeric strings, which are the single worst case for speech recognition. Policy numbers, IBANs, order references, postcodes, registration plates. There is no linguistic context to help the model, the confusion sets are dense, and getting one character wrong invalidates the entire interaction. The catalogue of operational failure modes teams hit moving voice AI from pilot to production puts accent and dialect recognition bias alongside speech hallucination and escalation failure as the recurring causes of production breakdown.

The mitigations are unglamorous and they work. Boost domain vocabulary and product names into the recognition layer rather than hoping the model has seen them. Use constrained grammars or spelling alphabets for alphanumeric capture rather than free transcription. Read back anything consequential and get an explicit confirmation. Treat low recognition confidence as a routing signal, so a caller the model is struggling to hear reaches a human quickly instead of after four failed attempts. And measure word error rate on your own recorded calls, segmented by accent and channel, because a vendor's benchmark tells you almost nothing about your traffic.

How Do You Test Something That Only Exists as Audio?

Ask an engineering team how they test their voice agent and you will usually get one of two answers. Either they call it themselves before each release, or they have unit tests over the text layer and nothing at all over the audio layer. Both mean the same thing in practice: nobody knows whether the change they shipped on Tuesday made the agent worse, and they will find out from a customer complaint or a containment metric that drifts over a fortnight.

Testing at the text layer is necessary and it covers a minority of the risk. It cannot tell you that the new synthesis voice speaks numbers too quickly for callers to write down, that the endpointing change now cuts off anyone who pauses to think, that the interruption handler leaves the conversation state divergent, or that the recognition model regressed on one regional accent after a routine version bump. Every one of those is a real incident we have seen, and none of them are visible without evaluating actual audio.

A working harness has a specific shape. Build a corpus of recorded and synthesised callers spanning the accents, background conditions and edge cases you actually serve, then replay them against every candidate build. Include the adversarial cases deliberately: callers who interrupt constantly, callers who go silent, callers who change their mind mid-transaction, callers with a television on behind them, callers who try to talk the agent into something it should refuse. Score each run on task completion, on whether escalation happened when it should have, on word error rate, on barge-in recovery, and on the latency distribution rather than the mean, because the mean hides exactly the tail that ruins calls.

Expectations should be set against a ramp rather than a launch number, which is the piece most business cases get wrong. The benchmark ranges in the evaluation guide cited above describe containment climbing from roughly 40 to 60% in the first month to 75 to 85% by month six on mature systems, with barge-in recovery above 90% as the healthy target. A voice agent that contains half its calls in week two is on a normal trajectory. If your business case assumed 80% from launch, you will kill a working system three months before it starts paying for itself.

One more metric deserves elevating above the rest: correct escalation. An agent that hands a caller to a human at the right moment, with the context already gathered, is delivering most of the value with none of the reputational risk. An agent optimised purely for containment learns to hold onto calls it cannot resolve, which is the worst outcome available to you and the one your containment dashboard will report as success.

What Does a Minute of Conversation Actually Cost?

Voice is the one AI application where unit economics are genuinely knowable in advance, which makes it unusual and worth exploiting. You know your call volume, you know your average handle time, and every component of the stack is priced per minute or per token. The mistake is not that teams cannot do the arithmetic. It is that they do it on the wrong denominator.

The component picture is roughly this. Carrier telephony runs at fractions of a cent per minute. Speech recognition sits in the low cents. Synthesis is usually the most expensive single component. Model inference depends heavily on how much context you resend each turn, which in a long call is a great deal more than teams expect. The worked cost model for voice agents published this year puts raw assembled stacks well below managed platforms, with the platform premium reflecting the orchestration, session infrastructure and telephony you would otherwise operate yourself. Both numbers are small relative to a human handling the same call. Neither is the number that decides your business case.

The number that decides it is cost per resolved contact, and it behaves very differently. A call the agent handles end to end is cheap. A call the agent mishandles costs you the agent minutes, the human minutes that follow, the caller's patience, and frequently a second inbound call. At a 70% containment rate, the 30% that escalate are paid for twice. That is why the escalation quality work in the previous section is an economic argument rather than a customer experience nicety, and why a system tuned to maximise containment can raise your cost per resolution while improving every number on the dashboard.

Two smaller traps are worth naming because they show up on the second invoice rather than the first. Context growth is one: naive implementations resend the entire conversation each turn, so a twelve-minute call costs disproportionately more than four three-minute calls, and nothing in your per-minute model predicted it. Silence is the other: an agent that stays on the line while a caller looks for their reference number is burning inference and synthesis capacity to produce nothing. Both are ordinary engineering problems with ordinary engineering fixes, and both are invisible until you instrument cost per call rather than cost per minute.

What Does the Law Require Before the Agent Says Hello?

On 2 August 2026, the transparency obligations in the EU AI Act became applicable. If you are running a voice agent to European callers, this is not a future compliance item on a roadmap. It applied seventeen days before this article was published.

The text of Article 50 of the AI Act requires that AI systems intended to interact directly with natural persons are designed so that those persons are informed they are interacting with an AI system, unless that is obvious to a reasonably well-informed and observant person in the circumstances. For a voice agent good enough to be worth deploying, it is by definition not obvious, so the disclosure is required. The practical reading is narrow and unforgiving: the disclosure must be spoken, it must come at the start of the interaction, and burying it in terms and conditions or naming the system an assistant does not discharge the obligation. The Cloud Security Alliance research note on Article 50 transparency sets out the same reading, and puts exposure for transparency breaches at up to 15 million euros or 3% of worldwide annual turnover.

If you also call the United States, a separate regime applies and it predates the AI Act. In February 2024 the Federal Communications Commission adopted a declaratory ruling confirming that the Telephone Consumer Protection Act applies to AI technologies that generate human voices, which means AI-generated voices count as artificial for TCPA purposes and outbound calls using them require the prior express consent of the called party. Inbound support automation is a different matter, but any outbound campaign is now squarely inside a statute with a well-developed private right of action.

Then there is the data. Call audio is personal data, recordings need a lawful basis and a retention policy, and if you use voice characteristics to identify or verify a caller you are processing biometric data under GDPR, with the additional conditions that attracts. Where the audio is transcribed, where inference runs, and which sub-processors touch the stream are all questions your data protection officer will eventually ask, and the answer that a US-hosted API processes every call is a conversation you would rather have before launch than after.

None of this is a reason not to build. It is a reason to build the disclosure, the consent capture, the retention policy and the region pinning into the first version, because all four are cheap at design time and genuinely expensive to retrofit into a system that has already taken a hundred thousand calls.

Who Should Actually Be Building This?

Look at the disciplines above and the staffing answer becomes uncomfortable. You need someone who understands real-time media transport, because the audio path is WebRTC or SIP and both fail in ways application developers have never debugged. You need someone who thinks in distributed state, because interruption handling is a cancellation and reconciliation problem. You need someone who can build and maintain an evaluation harness over audio. You need someone who knows enough about telephony to reason about codecs, jitter buffers and carrier behaviour. And you need someone who can hold the compliance obligations in their head while making product decisions.

That is not a prompt engineer. It is not a generalist full-stack developer with a weekend of enthusiasm either, and the market for people who genuinely have this combination is thin, because the discipline is roughly two years old at production scale and its component skills were previously spread across telecoms, streaming media and machine learning. Hiring it into a permanent in-house team in Western Europe is possible and slow, and most organisations discover the timeline after they have already committed to a launch date.

There is a structural point underneath the scarcity that matters more than the scarcity itself: this work does not end. Speech models get deprecated and replaced on vendor timelines you do not control, and a routine version bump can move your word error rate on one accent. Carriers change behaviour. Callers learn what your agent can do and start asking for more. Your product changes and the agent has to change with it. A voice agent is not a project with a completion date, it is a system with an operating cadence, and the failure pattern is depressingly consistent: an agency builds a good one, the engagement ends, the eval suite stops being run, and eighteen months later containment has quietly decayed and nobody can say when it started.

That combination, scarce senior skills plus permanent ownership plus a need for fast feedback loops, is the shape a dedicated nearshore team fits well. At Stepto we staff this kind of work as long-running dedicated teams rather than fixed-scope projects, with engineers in Serbia working the same hours as clients in Western Europe, the UK and the Nordics. Overlapping hours matter more here than in almost any other discipline we work in, because the person who noticed the agent mishandling calls this morning needs to sit with the engineer and listen to the recording today, not describe it in a ticket that gets picked up eleven hours later. Real-time systems are debugged by listening to them together.

Working inside the EU legal perimeter matters for the same practical reasons. Article 50 disclosure, GDPR lawful basis for recordings, biometric processing conditions and data residency for audio are live constraints on the architecture rather than a compliance review at the end, and a team that is subject to the same regime as you starts from the same defaults. If you are weighing this against building the capability in-house, the honest framing is not rate against rate. It is whether you can hire four scarce specialists inside your launch window, and whether you can keep them interested in operating the system in year two, which is when it either compounds in value or quietly rots.

How Would You Scope a First Deployment?

The instinct is to replace the whole phone queue, and it is wrong for the same reason big-bang migrations are always wrong: you learn nothing until everything is at risk simultaneously. What works is a single narrow intent, chosen for volume rather than for how impressive it sounds in a steering committee.

Pick the call type that arrives most often and has the clearest definition of done. Order status. Appointment booking or rescheduling. Balance enquiry. Delivery slot changes. High volume gives you a statistically meaningful signal within weeks instead of quarters, and a crisp completion criterion gives you an unambiguous task success metric. Resist the open-ended assistant that can help with anything, because you cannot measure it, cannot test it, and cannot tell whether it is getting better.

Instrument before you launch, not after the first incident. Every call should produce the audio, the transcript, per-turn latency traces broken down by pipeline stage, recognition confidence, the tool calls made, and the outcome. Teams that skip this spend their first month arguing about whether the agent is any good, because nobody has the data to settle it. Teams that do it can answer the question in an afternoon and spend that month improving the thing instead.

Build the evaluation corpus before the agent, not alongside it. Fifty to a hundred recorded or synthesised calls covering your accent range, your noise conditions and your awkward cases is a modest piece of work that becomes the asset the entire programme runs on. Every subsequent change gets measured against it, and it is the only mechanism that will catch a silent regression from a vendor model update.

Then ship narrow and keep the exit obvious. Route a small share of live traffic, keep a human one clear request away at all times, and treat correct escalation as a success in your metrics rather than a failure. Expect containment in the 40 to 60% range early and plan the business case against a six-month ramp. And decide who owns the system in year two before you launch it, because the eval suite that nobody runs is exactly how a working voice agent becomes an embarrassing one.