Voice agent latency budget
Everything between a caller finishing their sentence and hearing the first syllable back. Most teams blame the language model. It is usually the endpointing.
Start from a stack
What you get before anyone profiles it.
1.51 s
At 1510 ms callers start speaking over the agent, which costs you a turn and usually a re-prompt. Endpointing alone is 40% of it.
The budget, as numbers
| Hop | Time | Share |
|---|---|---|
| Network round trip | 140 ms | 9% |
| Endpointing (waiting for silence) | 600 ms | 40% |
| Speech-to-text finalisation | 200 ms | 13% |
| Routing / classification | 0 ms | 0% |
| Language model, first token | 450 ms | 30% |
| Speech synthesis, first audio | 120 ms | 8% |
| Total | 1510 ms | 100% |
The point of this tool: load “Typical cascaded” and drag endpointing from 600 ms down to 250 ms. You just saved more time than switching to a faster language model would have, and it cost you nothing but a config value. Then compare the two routing presets — the same classification step costs 100 ms on a typed decision model and 400 ms on a chat model.
Why endpointing dominates
Every other hop is work. Endpointing is waiting. You are holding the line open, doing nothing, because you cannot yet prove the caller has finished their sentence. Set it short and the agent talks over people who paused to think; set it long and every single turn pays the penalty. Most SDKs ship a conservative default somewhere between 500 and 800 ms and most teams never touch it, which means the majority of their latency budget is spent on silence. Semantic turn detection — deciding from the words whether the thought is complete, not just from the pause — is the way out, and it is worth more than any model upgrade on this list.
What each band actually sounds like
| Response time | What the caller experiences |
|---|---|
| Under 800 ms | Indistinguishable from a human on a slightly slow line. Nobody comments on it. |
| 800 – 1,200 ms | A noticeable beat before every reply. Works, but the call feels stilted and long conversations tire people. |
| 1.2 – 2 s | Callers start filling the gap — “hello?” — and now you are handling barge-in on every turn. |
| Over 2 s | Assumed to be a dropped call or a broken system. Hang-up rates climb sharply. |
Assumptions
- This measures time to first audio, not time to a complete answer. If you wait for the full response before speaking, add the whole generation.
- Every hop assumes streaming. A single non-streaming stage invalidates the model — that is what the cross-region preset is really showing.
- Speech-to-speech collapses transcription, reasoning and synthesis into one number under the language model. It wins on latency and loses on cost, control and per-layer failover, which is the trade covered in the cost calculator.
- Network figures assume the media path terminates in the same region as your agent. Split those and the round trip doubles.
Related
- Why voice agents feel slow — the longer argument behind this tool.
- A System One model for the routing hop — where that 100 ms routing number comes from.
- Voice-agent cost calculator — the other half of the problem.