LLMs & AI Agents · Voice agents
Voice agents in production: the architecture decisions that matter
A demo takes an afternoon. Whether real callers accept the agent is settled by decisions made early in the architecture: the response time target, how the models are arranged, turn taking, and what you buy against what you keep.
Julia Rose
Key takeaways
- People take turns after about 200 milliseconds. The response time target belongs to the use case and should be set before the architecture is chosen.
- Measure on the caller’s side and look at the tail. If one turn in twenty is slow, two calls in three contain an awkward silence.
- Cascade, speech to speech or hybrid decides how much control, auditability and vendor independence you keep.
- Transport, turn detection and telephony are worth buying. The latency budget, conversation design, evaluations and swappability are not.
- The common production failures do not sit in the model, so a better model does not fix them.
Response time is a requirement
A voice agent demo takes an afternoon: speech to text, a language model, text to speech. Getting real callers to accept one is a different problem. In production a voice agent is a real time system in which audio transport, turn taking, models, external tools and conversation design all compete for the same second. Most of the decisions that settle whether it works are made early in the architecture, and they are expensive to change later.
In conversation, people take turns after about 200 milliseconds. Half a second is noticeable, and longer pauses quickly feel like hesitation or a dropped line. How much delay is acceptable depends on the case. An entertainment avatar at an event can afford more than an agent handling a booking under time pressure. What matters is that the target is set deliberately, per use case, before the architecture is chosen.
Two things make that target meaningful. It has to be measured end to end, from the caller’s last word to the first sound of the answer, on the caller’s side. And it has to look at the slow moments rather than the average. Engineers use the 95th percentile, written p95: if one turn in twenty is slow, about two in three calls with twenty turns contain at least one awkward silence. Model generation time, the number vendors like to quote, leaves out transport, end of turn detection and tool calls.
Speech recognition, language model and speech output are largely commodity and can be rented. The latency hides in everything around them, which is where the engineering effort goes.
Cascade, speech to speech or hybrid
The first architectural choice is how many models sit in the conversation loop. Speech to speech models have matured quickly. They handle being talked over, they make listening noises while the caller speaks, and they keep tone and emphasis. Reasoning and tool calling are no longer the reason to avoid them. Control is.
| Cascade | Speech to speech | Hybrid | |
|---|---|---|---|
| Naturalness | Good, depends on tuning | Best | Close to speech to speech |
| Control per stage | Full | Low | High where it matters |
| Following strict instructions | Strong | Weaker | Strong in the backend |
| Auditability | Transcripts at every step | Needs separate transcripts | Transcripts in the backend |
| Vendor coupling | Low, stages swappable | One vendor holds the conversation | Medium |
| Good fit | Numbers, addresses, regulated domains | Open conversation, events, fast prototypes | Enterprise service with transactions |
In the hybrid pattern a small, fast model owns the conversation and keeps the rhythm, while anything that needs accuracy, policy or a system of record is delegated to larger models through tool calls. The caller hears one continuous conversation, and the guardrails run in the text layer, where they are easier to get right. Whatever the architecture, keep the transcripts. Evaluation, audits and every later improvement depend on them.
Where it runs, and what to buy
Physical location is a model choice. Picking the best speech recognition, the best language model and the best speech output independently often produces a stack spread across providers and regions, and every hop between them costs more than the better models save. Co-located models that are good enough usually beat distributed excellent ones. In the EU, where the data may sit often decides the region before latency does, so it belongs in the first architecture discussion.
The connection itself needs a streaming protocol built for audio: WebRTC for browsers and apps, SIP for phone lines. Request and response over HTTP adds delay on every turn and makes interruptions close to impossible. This is plumbing nobody should build from scratch. Frameworks such as LiveKit or Pipecat and the managed platforms provide it. Buying that plumbing is sensible. Handing over how the system behaves is not.
| Worth buying | Keep in your own hands |
|---|---|
| Audio transport, sessions, telephony | Response time targets, measured on the caller’s side |
| Turn detection and interruption handling | Conversation design and prompts |
| Streaming between the models | Evaluations and the call recordings behind them |
| Noise suppression and per stage metrics | The option to swap recognition, model and voice separately |
| Where the data sits and what callers are told |
Bundling everything with one vendor is the fastest way to a running system. The price is an architecture that is hard to change when a better model appears in six months.
Turn taking is a product decision
Deciding when the caller has finished is the hardest part of the loop and the most underestimated. Silence is a weak signal. People reading out a reference number, correcting themselves or searching for a word in a second language pause for more than 700 milliseconds mid sentence. A fixed silence timeout either cuts them off or adds dead air to every turn. At 800 milliseconds on each turn of a thirty turn call that is 24 seconds of silence.
That timeout is a policy about how your callers speak, and it should differ by workflow: patient while capturing an address, quick for a yes or no question. Production systems combine voice activity detection, the recogniser’s end of utterance signal and a small model that predicts from wording and intonation whether the thought is finished. The detectors are small and inexpensive. The policy around them makes the difference.
Non-native speakers pause the longest, so a threshold tuned on native speakers in a quiet office cuts off a large share of real callers. Audio quality sets the ceiling for all of it. Microphone choice, echo cancellation and noise suppression come before any model, especially for avatars and kiosks in public spaces.
Where voice agents break in production
Most failures do not show in a demo. These are the ones that turn up with real callers.
| Failure | What it looks like | What prevents it |
|---|---|---|
| Server side metrics only | Dashboards green, callers unhappy | Timings per step measured on the caller’s side, with recordings kept next to each trace |
| Interruptions half handled | The agent later confirms details the caller never heard | Interruption handling that stops the answer and keeps only what was actually played |
| Tools treated as free | Silence during lookups, callers think the line dropped | An acknowledgement, a timeout and a clear owner for every backend call |
| Text only testing | Tests pass while callers hang up | Tests on real call audio, false cut-offs tracked as a number, synthetic calls at scale |
| Chat prompts on a voice channel | Lists, links and long sentences read aloud | Prompts written for speech, explicit conversation states, spoken confirmation before any action |
None of these is a model problem, so none of them is solved by switching to a better model.
Why work with a partner, and why us
A strong internal team can take the points above and build on a platform directly. A partner pays off in the parts a platform does not solve: fitting the agent into an existing system landscape, meeting European data requirements, and designing conversations that hold up with real users in real environments.
- Integration into existing systems. As part of the Apollogic Group, an SAP and Microsoft partner, we connect voice agents to the ERP, CRM and service systems they have to read from and act in.
- European data requirements. Offices in Hamburg and Poznań, hosting in the EU, on your own servers where a project needs it, and experience in regulated settings such as hospitals.
- Multilingual by default. German, English and Polish in the team, and testing with non-native speakers built into every project.
- Vendor neutral. We do not sell a platform, so we choose the components that fit the case and keep them replaceable.
The situations we usually work in: interactive avatars for events and trade fairs, where open conversation in loud halls matters more than strict accuracy; self-service avatars at customer-facing locations, with speakerphone audio, multilingual users and transactions that have to reach a backend system reliably; and assistants on top of enterprise systems, where answers and actions come from company data and every step has to be traceable.
What a project looks like
There is no single right setup, so we do not start from a template. A voice project with us runs in three phases, with a decision point after the first.
- Analysis. We look at the whole flow: the scenario and where conversations go off track, the setting, the systems the agent has to reach, and how accurate the answers need to be. An entertainment avatar can improvise, a system handling bookings cannot. The result is an architecture, response time targets that fit the case and a scoped pilot. You decide whether to continue.
- Build and pilot. We build the agent, connect it to your systems and test it with real speech: different speakers, languages, background noise and interruptions, not a quiet meeting room.
- Go live and operation. Rollout with monitoring of every step, regular review of real calls, and components that can be swapped when better models appear.
The questions buyers usually ask belong in the analysis: how the agent connects to an existing contact centre or phone system, which languages it has to support, where recordings are stored and who can reach them, how callers are told that they are speaking to an AI, as the EU AI Act requires, and who runs and improves the system after launch.
If you are starting from a use case, comparing providers or sitting on a prototype that does not hold up with real users, that is the point where we usually come in. How we scope projects is on How we work.
theBlue.ai is an enterprise AI company with offices in Hamburg and Poznań. The team has been working on AI projects since 2016 and belongs to the Apollogic group with more than 120 developers. More about us on About us.
Sources: Stivers et al., “Universals and cultural variation in turn-taking in conversation”, PNAS 106(26), 2009 for the gap between speakers. Cekura, August 2026 for the distribution of response times across six platforms. Parts of this article come from the talk “Voice Agents: Where Latency Hides and How to Fight It”, which our CTO Agata Chudzińska gave on 29 September 2026 at ML Con in New York.
Planning an AI project?
In the process analysis we map your workflows, rank the possible use cases by return and propose an architecture.
Request a process analysisFrequently asked questions
The benchmark is the human rhythm: between two people the gap averages about 200 milliseconds (Stivers et al., PNAS 2009). Half a second is noticeable and at two seconds callers hang up. The target should be set per use case and measured on the caller’s side, from their last word to the first sound of the answer.
A cascade of recognition, language model and speech output gives control over every stage and a transcript at each step, which suits numbers, addresses and regulated domains. Speech to speech sounds more natural and suits open conversation and events, but gives up control. The hybrid lets a fast model run the conversation and hands anything precise to larger models.
Audio transport, sessions, telephony, turn detection and noise suppression are worth buying. Keep the response time targets, the conversation design, the evaluations with their recordings, and the option to swap recognition, model and voice separately.
The common reasons sit outside the model: metrics taken on the server instead of the caller’s side, interruptions only half handled, tool calls with no acknowledgement, tests that only ever use text and never real call audio, and prompts written for chat. A better model fixes none of them.
Turn taking is the decision about when the caller has finished speaking. Silence alone is not enough, because people pause for more than 700 milliseconds mid sentence. Production systems combine voice activity detection, the recogniser’s end of utterance signal and a small model that predicts from wording and intonation whether the thought is finished.
Julia has been part of theBlue.ai since 2019 and has accompanied the development of AI applications in the enterprise environment since the company’s early days. In her role as Marketing Lead, she works closely with the engineering and consulting teams and makes complex technical topics understandable and accessible for decision-makers.
In her articles, she writes about practical experience from enterprise AI projects, as well as the challenges and opportunities of using AI in companies.