Your Agent's Real Failure Rate Isn't in the Ticket Queue

Complaint Rate Measures Anger, Not Accuracy

A complaint is a two-step event. The user has to notice the answer is wrong, and then care enough to do something about it. Both steps filter hard. For anything the user asked precisely because they didn't already know it — which is most of what you'd build an agent for — step one barely fires. They can't verify the answer, so a confident wrong answer and a confident right answer look identical.

That's the trap with fluent output. LLMs produce plausible prose at the same confidence whether they're right or hallucinating, and we've written before about why self-reported confidence scores don't rescue you here. The user has no signal to distinguish the two, so the wrong answer sails through unreported.

Across our engagements, when a team quotes a complaint rate as their quality number, it's usually off by a large multiple from what sampled review finds. Complaint rate tracks the ceiling of your most engaged, most expert, most annoyed users. It tells you nothing about the median interaction.

A confident wrong answer and a confident right answer look identical to the user who asked precisely because they didn't know.

The Two Silent Failures That Cost You Most

There are two shapes of silent failure and they hurt in different ways. The first is quiet disengagement: the user gets a bad-enough answer, decides the tool isn't worth trusting, and stops coming back. You don't see a complaint. You see a churn number three months later that nobody connects to the agent. Adoption flatlines and everyone blames onboarding.

The second is worse: the accepted bad answer. The user can't verify it, so they trust it and act on it. In a support context that's a customer given wrong billing guidance. In an internal knowledge agent it's an analyst pasting a fabricated figure into a board deck. These don't just fail to help. They inject wrong information into your business, dressed up with a veneer of authority.

Neither shows up where you're looking. The disengaged user simply stops appearing in the data — there's no event to log when someone walks away. The trusting user is the tricky one: their session looks like a clean, successful interaction right up until the damage surfaces somewhere you'd never think to connect it.

Shadow Evals on Real Traffic, Not Your Golden Set

Your pre-launch eval set tells you how the agent does on the questions you thought to write down. Real traffic is a different distribution — messier phrasing, edge cases you never imagined, questions outside the agent's actual competence. So the first instrument we stand up is a shadow eval that runs graders against a sample of real production traffic, continuously. Not the questions you imagined. The ones people actually ask.

For every task type where a machine-checkable ground truth exists — retrieval hit-rate, factual claims that resolve against a source, format and policy compliance — we grade automatically. Retrieval is where we always start, because if the right passage never got retrieved the answer was doomed before generation, and a retrieval-only eval catches that class of failure cheaply. LLM-as-judge fills gaps for softer dimensions, but only after we've calibrated the judge against human labels so we know its own error rate.

The point isn't a single quality score. It's a failure rate estimate on the distribution your users actually send, refreshed continuously, so drift shows up as a trend and not as a surprise churn number. This is also how you catch the agent that passed every eval and rotted anyway once prompts and vendor models started shifting underneath it.

Sampled Human Review Is Non-Negotiable — But It's Cheap

Automated grading can't cover everything, and every team assumes human review means reviewing everything, which is why they never start. You don't have to. A stratified random sample gets you a defensible estimate with a confidence interval attached.

The discipline that matters: stratify by task type and by risk, not uniformly. Sample more heavily where a wrong answer is expensive or unverifiable, less where it's cheap and self-correcting. Have reviewers grade against a rubric that separates 'wrong' from 'unhelpful' from 'refused when it shouldn't have,' because those need different fixes. And rotate the samples so reviewers can't pattern-match their way to rubber-stamping — the same failure mode we've written about in human-in-the-loop gates that train reviewers to click yes.

A few hundred reviewed interactions a week, stratified properly, will tell you your silent failure rate to within a few points. That's not a research project. That's one reviewer, part-time, with a decent tool.

• →Stratify samples by task type and blast radius, never uniformly.
• →Grade against a rubric that separates wrong, unhelpful, and over-refusal.
• →Rotate samples so reviewers can't fall into rubber-stamping.
• →Attach a confidence interval to the estimate — it's a measurement, treat it like one.

Behavioural Signals Catch What Grading Misses

Users leak their dissatisfaction through behaviour even when they never complain. The loudest signal is the re-query: a user gets an answer, then immediately rephrases the same question. That first answer missed. Track semantic similarity between consecutive queries in a session and you've got a cheap, high-recall detector for the answers that aren't landing.

Abandonment is the mirror image. The user asks, gets an answer, and does nothing — no follow-up, no action, no copy. In a support agent, the session that ends in silence and then reappears as a human ticket an hour later is a deflection that failed quietly. Session-end-then-escalate is one of the most honest quality signals you have, and it never touches your complaint queue.

None of these are ground truth on their own. A re-query might be a follow-up, not a correction. But as leading indicators, layered on top of sampled review, they let you triage which interactions to pull for human grading — turning your review budget toward the sessions most likely to contain a real failure. Instrumenting this properly needs span-level traces you actually captured before go-live, not logs you wish you'd added after the fact.

Be Honest: Most Teams Have No Idea What This Number Is

Ask a team running a production agent what their silent failure rate is and you'll mostly get a confident guess anchored to complaint rate, which we've established is the wrong instrument. That's not incompetence. Nobody set up the measurement because the failures are, by definition, invisible, and green dashboards feel like evidence.

The honest part: even with all three instruments, you get an estimate. LLM judges have their own error rates. Behavioural signals are noisy. Human review is a sample. What you're buying is a defensible number with a confidence interval, and the ability to watch it move when you change a prompt or a model version.

The remaining hard problem is the accepted-wrong answer where no behavioural signal fires and the ground truth is genuinely expensive to establish. There's no clever telemetry that fully solves it. The only real lever is constraining the agent's scope so it refuses or escalates on the questions it can't answer reliably, which is why where you draw the refusal boundary does more for real quality than another round of prompt tuning. Start by pulling two hundred random production interactions and having a human grade them this week. Whatever number comes back, it's more truth than your ticket queue has ever given you.