Budget Latency Across the Whole Agent Path, Not Just the Average

Users Remember Your Worst Response, Not Your Median

Median latency is the number that makes everyone feel good in the standup. It's also close to useless for predicting whether people trust the thing. A support agent that answers in 3s ninety-nine times and hangs for 25s once teaches the user one lesson: this thing sometimes just stops. After that they hover over the tab. They don't multitask, they wait, they lose faith. That one bad experience overwrites the ninety-nine good ones.

In our work the pattern is consistent — the agent that averaged 4s and felt broken had a p50 of 2.9s and a p99 north of 20s. Nobody was optimising the p99 because the average looked defensible. But an average is a lie told about a distribution. Two agents can share a p50 and feel completely different: one is tight around the median, the other has a fat right tail that shows up exactly when a tool call retries or retrieval hits a cold index.

The fix starts with instrumenting the tail, not the mean. If your dashboard shows p50 first and you have to click twice to find p99, you're going to optimise the wrong number by default. Put p95 and p99 on the front page. Alert on them. Track them per stage, not just end-to-end — the span-level tracing you wire in before go-live is what tells you which stage owns the spike.

An average is a lie told about a distribution.

Treat Latency as a Budget You Allocate, Not a Number You Discover

Before you build, decide what the total budget is and split it across the path. If your target is "first token in under 1.5s, full answer under 6s at p95," that number has to survive being divided among retrieval, tool calls, model inference, and streaming overhead. When you write it down as an allocation, the over-runs become obvious.

A rough budget for a RAG-plus-tools agent might look like this. Retrieval gets 400ms. Any tool call gets 800ms with a hard ceiling. Model time-to-first-token gets 700ms. Everything else — orchestration, serialisation, network — gets 300ms of slack. Now every stage has a number to defend, and when retrieval quietly creeps to 1.2s because someone changed the chunking, the budget flags it instead of the p50 absorbing it silently.

The stages that blow budgets are rarely inference itself. They're the sequential tool calls an agent makes when it second-guesses itself, and the tool-call sprawl where a six-call job balloons to forty. Tail latency and tool thrashing are the same problem viewed from two angles: uncontrolled fan-out. Cap the calls, cap the tail.

• →Write the total latency budget as a p95 target before the build, and split it per stage.
• →Give every tool call a hard timeout ceiling — an unbounded call is an unbounded tail.
• →Track each stage's contribution to p99 separately; the culprit is almost never the model.
• →Cap agent tool-call fan-out — sequential retries are the single biggest tail driver.

Streaming Changes Perceived Latency More Than Any Backend Fix

The cheapest, largest win we've shipped on latency wasn't a backend change at all. It was streaming partial output. A user staring at a spinner for 5s and a user watching tokens appear at 1.2s and finish at 6s have wildly different experiences of the same total wall-clock time. Perceived latency is what governs trust, and dead air is the enemy.

The agent that felt broken had no streaming. It computed the full answer server-side and returned it in one blocking response. Even when it was fast, it felt slow, because the user had no signal anything was happening. We added token streaming and a status line — "searching the knowledge base," "checking the policy doc" — and the complaints dropped before we'd shaved a single millisecond off the actual compute.

Stream more than tokens. Stream progress. If retrieval takes 400ms, say so on screen. If a tool call is running, show it. The goal is that the user is never looking at a static screen wondering whether the thing crashed. A visible slow answer beats an invisible fast one for trust every time.

Speculative Prefetch and Timeout-and-Fallback Are the Tail Killers

Two techniques move p99 in ways that median-focused tuning never will. The first is speculative prefetch: start the likely-next work before you're certain you need it. If a user's message almost always triggers a retrieval, fire the retrieval in parallel with the intent classification instead of waiting for it to finish. You waste some work on the requests where you guessed wrong. But you shave a full serial stage off the ones where you guessed right, and on a path with three sequential stages, collapsing two of them into parallel is often the difference between a 6s and a 4s p95.

The second is timeout-and-fallback, designed deliberately rather than bolted on after an incident. Every stage that can hang needs a timeout and a defined thing that happens when it fires. If retrieval doesn't return in 800ms, do you answer without it or degrade to a narrower response? Decide before production decides for you. An agent that returns a slightly worse answer in 4s beats one that spends 22s on a perfect answer the user already abandoned.

The uncomfortable part is that fallback design forces you to admit your accuracy target and your latency target are in tension. You can't have both at the p99. Pick which one degrades under pressure, and make it explicit.

• →Speculative prefetch: run likely-next stages in parallel, accept wasted work on wrong guesses.
• →Give every hangable stage a timeout and a defined fallback — cache, narrower answer, or graceful degradation.
• →Decide up front whether accuracy or latency degrades under load; you can't protect both at p99.
• →Test the fallbacks under injected failure, not just the happy path — this belongs in your [pre-launch stress harness](/insights/stress-testing-ai-agents-before-launch).

The Accuracy-Responsiveness Trade-off Is Real, and You Have to Pick

More retrieval, more reranking, more reasoning steps, a bigger model — each buys accuracy and each costs latency. The teams that get stuck are the ones pretending they can max both. They add a reranker to fix a retrieval miss, add a verification pass to catch an answer that overreaches past its cited passage, add a second model call to double-check, and the p99 quietly walks past 15s while every individual change looked justified.

Set the responsiveness floor first, then spend your accuracy budget inside it. If first token has to land in 1.5s, a verification pass that adds 900ms to time-to-first-token is off the table — you run it after streaming starts, or you don't run it inline at all. This ordering discipline is what keeps well-intentioned quality work from destroying the feel of the product.

For a lot of tasks the honest answer is that you don't need agency at all — a constrained workflow with fixed stages is both faster and more predictable at the tail than a model deciding its own path. When latency matters, taking agency away from the model is often the responsiveness fix nobody wants to hear.

What's Still Hard, and Where to Start

None of this makes the tail disappear. Third-party model APIs have their own p99 spikes you don't control, and when a vendor has a slow minute, your fallback design is the only thing standing between that and your users. Cold caches and index rebuilds reintroduce tail latency you thought you'd killed. Be honest about it: p99 is a number you defend continuously. You never clear it once and move on.

Start somewhere embarrassingly simple: pull your p99 per stage and look at it. Most teams have never seen it broken down that way. When you do, the fix usually names itself — one stage owns the tail, and it's almost never the one you expected. Then add streaming. It buys you perceived-latency headroom while you fix the real thing underneath.

If you're shipping an agent people rate on feel rather than on benchmarks, budget the latency the way you'd budget a financial plan — per line item, defended, with a stated fallback when a line over-runs. The average will take care of itself. The tail won't.