The Trace You Wish You Had After an Agent Fails in Prod

Observability Bolted On After an Incident Is Already Too Late

The usual sequence: an agent ships, works in the demo, and runs quietly for a few weeks. Then a customer gets a confidently wrong answer, it escalates, and someone opens the logs. What they find is a prompt, a response, and a timestamp. No record of which documents were retrieved, whether the retrieval was even relevant, which tools fired, or why the model chose the words it did.

So the team spends a day trying to reproduce the failure, usually failing because the index has changed or the input was slightly different. They add tracing in response to the incident. The next incident is different, and the tracing they added doesn't cover it either. This is the pattern we see most often when we come into an audit — detecting agent failure in production is impossible when the telemetry was never designed to explain a single request.

The fix isn't more logging volume. It's structured, causal telemetry attached to every request before it ships, so the first incident is debuggable rather than a forensic dig.

The trace you need to debug a failure has to already exist when the failure happens.

What a Real Agent Trace Contains

We model every request as a trace with a span per meaningful step. For a RAG agent that's usually query rewrite, retrieval, rerank, tool calls, and generation, plus any post-generation guardrail. Each span carries a start and end time. That way per-step latency is a first-class attribute you read straight off the span, instead of piecing it together from the order your logs happened to land in.

The spans are the skeleton. The attributes are what actually let you localise a fault. On the retrieval span we attach the rewritten query, the retrieved document IDs and their scores, and the index version. On tool spans we record the arguments, the result size, and whether it errored or timed out. On the generation span we attach token counts in and out, the model and prompt version, and — critically — the refusal reason if the model declined or hedged.

Refusal reasons and doc IDs are the two attributes teams almost never capture and almost always need. If you know which document IDs came back, you can answer the single most useful diagnostic question in RAG: did the model hallucinate, or did retrieval never surface the right passage? That distinction, which we've argued deserves its own retrieval-only eval before you grade answer quality, is invisible without the doc IDs on the trace.

• →Retrieval span: rewritten query, retrieved doc IDs + scores, index version, latency
• →Tool spans: arguments, result size, error/timeout flag, latency
• →Generation span: model + prompt version, tokens in/out, refusal reason, latency
• →Trace-level: request ID, user/tenant, total latency, total cost, final outcome

A Failed Request, Localised in Minutes

Here's an anonymised trace from a support agent that returned a wrong policy answer. The user asked about a refund window. The agent confidently quoted 14 days; the correct answer was 30.

The trace showed six spans. Query rewrite looked fine — it expanded "refund window" sensibly. The retrieval span was where it broke: the top three doc IDs all pointed to an old version of the refunds policy, scores clustered around 0.82, and the index version attribute showed a build from three weeks earlier. The current 30-day policy document existed in the source system but had never made it into the index. The generation span was doing exactly what it should — grounding faithfully in what it was given. The refusal reason was empty because the model had no reason to refuse; the retrieved text read as authoritative.

Total time to localise: about four minutes, most of it spent confirming the index build date. Without the doc IDs and index version on the trace, this looks like a hallucination and you burn an afternoon tuning prompts. With them, it's obviously a stale index. That's a data pipeline job for the pipeline team. That's the whole point of the attributes: they tell you which team owns the bug.

Half the failures we diagnose this way trace back to the pipeline, not the model — the same data access layer that quietly stalls agent projects before they even reach production.

Off-the-Shelf Tracing vs What We Build

LLM-tracing tools — Langfuse, LangSmith, Arize Phoenix, the rest — are genuinely good at the model call. They give you nested spans, token counts, cost, and a decent UI for eyeballing prompts and completions. We use them and we recommend them. Start there.

Where they fall short is the boundary between your infrastructure and the model. Their instrumentation is strongest inside the LLM SDK and weakest at your retrieval layer and your permission checks — exactly where most production faults start. Auto-instrumentation hands you a generation span for free. But it treats your vector search as an opaque function call, so you don't get doc IDs, scores, or index version unless you go and attach them yourself.

What we build on top is the domain-specific attributes and the cross-boundary correlation. We propagate one trace ID from the API gateway through retrieval, tool calls, and any downstream service, so a slow answer can be pinned to a slow Postgres query rather than blamed on the model. And we wire trace attributes into the eval harness, so a regression in production surfaces as a diff against the golden eval set, not a vibe.

• →Use an off-the-shelf tracer for the LLM call — don't rebuild that
• →Add your own attributes at the retrieval and tool boundaries — doc IDs, scores, index version, permission decisions
• →Propagate a single trace ID end to end so latency and cost attribute to the right subsystem
• →Feed trace attributes back into evals so production failures become reproducible test cases

The Minimum Viable Observability for a Series A Team

You don't need a platform on day one. You need a trace ID and discipline about what you attach to it. A Series A team can get most of the value with structured JSON logs, one correlation ID per request, and a dashboard query — no procurement cycle required.

The failure mode to avoid is capturing everything and finding nothing. High log volume with no structure is as useless as no logs. Pick the attributes that answer a specific diagnostic question and attach those.

Here's the floor we'd insist on before any agent goes live:

• →A single request/trace ID that appears on every log line for that request
• →Retrieved doc IDs and their scores, plus the index version, on every retrieval
• →Tokens in/out and cost, per request, so a spend blowup is visible before the invoice
• →Per-step latency so you can tell a slow retrieval from a slow model
• →Refusal / fallback reason captured explicitly, not inferred
• →The prompt and model version that produced the answer, so you can tie it to a deploy
• →An alert on the two or three failure signals that actually matter — retrieval returning nothing, tool error rate, refusal rate spiking

What's Still Hard

Tracing tells you what happened. It won't tell you whether what happened was correct. A trace can show a clean retrieval, a sensible tool call, and a fluent answer that's still wrong because the source document itself was outdated. Observability localises faults; judging quality is a separate job. That's what evals are for, and the two have to be wired together.

Cost and cardinality are the other open problem. Full-fidelity traces on every request get expensive and noisy at scale, so you end up sampling — and sampling is exactly when the rare failure you needed to see gets dropped. We bias sampling toward requests that look risky: refusals, low retrieval scores, high tool-call counts, tail latency. It helps, but no one has fully solved keeping the interesting 1% while discarding the boring 99%.

The concrete next step is small: pick your next agent, and before it ships, make sure a single failed request can be explained from the trace alone. If it can't, you're not ready for go-live — you're ready for the first incident to teach you what you should have instrumented.