Catching Agent Failure in Production Before Your Users Do

The Failure Modes That Don't Throw Exceptions

Traditional software fails in ways your monitoring already understands — latency spikes, 500s, timeouts, OOM kills. Agents fail in a category your dashboards weren't built for: they succeed at the mechanical task and fail at the semantic one. The API returned 200. The JSON parsed. The answer was wrong.

We see four recurring shapes. Input drift, where the distribution of real requests slowly diverges from what you tested — a support agent starts getting a product line that didn't exist at launch. Retrieval drift, where the index goes stale or a chunk-boundary change quietly degrades relevance; we've written about why RAG demos break in production precisely because this one is invisible until you measure it. Confident fabrication, where the model invents a policy number or a citation and states it with total conviction. And tool-chain cascade, where one bad intermediate step — a mis-parsed date, a wrong entity resolution — poisons everything downstream while each individual step looks fine.

None of these throw an exception. That's the whole problem. Your uptime is 100% and your agent is wrong 8% of the time, and the two facts live in different systems that never talk to each other.

Your uptime is 100% and your agent is wrong 8% of the time, and the two facts live in different systems that never talk to each other.

Offline Evals Set the Floor, Online Monitoring Catches the Rest

A good eval harness is non-negotiable — we've argued that the eval harness is the actual deliverable for agentic work. But offline evals have a structural limit: they can only test failure modes you've already imagined. Production traffic is adversarial in the boring sense — it contains the inputs you didn't think of.

Online monitoring is how you catch the unknown unknowns. The mechanism is to sample live traffic, score it against reference behaviour or a judge model, and alert on movement. Not on absolute quality — on delta. A 4% drop in your relevance proxy week-over-week is a signal even if you can't immediately say why. This is also where golden datasets earn their keep, and where they betray you: golden eval sets rot within months as real traffic moves away from your fixtures, so the online signal is what tells you when to refresh them.

Concretely, the online layer we build usually includes:

• →A judge model scoring a sampled slice of production traffic on task-specific rubrics, run continuously, not in batch once a quarter.
• →Distribution monitors on inputs — embedding-space drift detection that flags when incoming requests move away from the training/eval distribution.
• →Cheap deterministic checks that don't need a model: schema validation, citation-exists checks, numeric range sanity, tool-output type checks.
• →A feedback capture path — thumbs, corrections, escalations — wired back to the same store as the automated scores, so human signal and machine signal live together.

Don't Build Your Human Gate on the Model's Own Confidence

The obvious design for deferral is: if the model is unsure, ask a human. The trap is using the model's self-reported confidence to decide. LLM confidence scores and raw logprobs are badly calibrated — a model will hand you a fabricated answer at 0.95 and a correct one at 0.6. We've dug into why LLM confidence scores lie and break your human-in-the-loop gate; the short version is that the number tells you how fluent the output was, not how likely it is to be right.

Build the gate on independent signals instead. Retrieval score below a threshold. Disagreement across an ensemble or across repeated samples. The absence of a supporting citation. A downstream deterministic check that fails. Any of these is a better trigger for deferral than the model's own certainty, because they don't share the failure mode you're trying to catch — you can't ask the thing that hallucinated whether it hallucinated.

The other half is making deferral cheap. If handing a case to a human costs the user two minutes and an apologetic email, your team will quietly raise the confidence threshold until nobody defers and the safety net is decorative. Design the human path as a fast queue with context already loaded, so it doesn't become a dead end.

The Telemetry Has to Exist Before the Incident

When an agent produces a bad output, the first question is always: what did it actually see and do? If you can't reconstruct the full trace — the raw input, the retrieved context, every tool call and its arguments and result, the intermediate reasoning, the final output, and the version of every prompt and model in play — you're debugging blind. And you cannot add this after the first incident, because the incident you care about already happened.

Trace everything, structured, from day one. Each request gets an ID that threads through every step. Every LLM call logs its prompt version, model version, temperature, token counts, and latency. Every tool call logs input and output. This is also what makes cost debugging possible — the agent that 10x'd its traffic and blew its budget was only diagnosable because the traces showed which step was fanning out calls.

A few heuristics we hold to:

• →One trace ID per request, propagated through every hop, so you can replay a single interaction end to end.
• →Log the inputs and outputs of tool calls, not just that they happened — the cascade failures live in the arguments.
• →Version prompts and models in the trace, so 'it got worse on Tuesday' maps to a specific change.
• →Store enough raw context to re-run the case offline — a production failure should become a new eval fixture within a day.

Contain the Blast Radius, Then Investigate

Detection without a containment reflex just means you find out faster that you're on fire. The agents we're most comfortable shipping have a graceful degradation path baked in: when the online signals cross a threshold, the system can automatically narrow its own autonomy — fall back to a more conservative model, route more cases to humans, disable a flaky tool, or in the worst case return a safe 'I can't answer this confidently' instead of a confident wrong answer.

This is the untrusted-insider mental model applied to reliability rather than security: assume the agent will occasionally act against your interests, and constrain what a single bad decision can do. An agent that can only draft, not send, has a bounded blast radius. An agent that can only recommend, with a human approving the action, has a smaller one still. The autonomy you grant should be earned by the monitoring you've proven.

The sequencing matters. Contain first — degrade autonomy, protect the user — then investigate with the traces you captured. Teams that try to root-cause during the incident, with autonomy still live, tend to make it worse.

What's Still Hard About This

Judge models are the load-bearing piece of online monitoring, and they come with their own calibration problems. A judge that drifts, or that shares a blind spot with the model it's scoring, hands you false confidence. We run periodic human audits of the judge's own scores to mitigate that, but it's an open problem. Nobody's solved it. Anyone selling you fully-automated quality monitoring with no human in the loop is selling you the thing that broke in the first place.

Drift detection on inputs is easier than drift detection on quality. You can measure embedding-space movement cheaply; measuring whether that movement actually hurt outcomes needs ground truth you often don't have in real time. And setting alert thresholds is genuinely hard — too tight and you drown in noise, too loose and you miss the slow degradations that matter most. Expect to tune these for weeks against real traffic.

The honest takeaway: you don't get to know your agent is safe. You get to shorten the time between it failing and you knowing. That's the whole game — instrument for detection, build the deferral and containment paths before you need them, and treat every production failure as a new eval case. Start by asking one question of any agent you have in production: if it started being wrong right now, how would you find out, and how long would it take?