How We Shipped a Support Agent That Deflected 40% of Tickets

The Baseline: A Queue That Was Drowning, and a Metric Nobody Trusted

The client was a Series B SaaS company with a support team of eleven handling roughly 3,000 tickets a month. Median first-response time had crept past nine hours. Their existing 'automation' was a canned-macro system and a help-centre search that customers routinely bounced off before opening a ticket anyway.

Leadership wanted deflection. The support lead wanted something subtler: to stop the team burning its best hours on the same forty questions — password resets, where's-my-invoice, how-do-I-export — while the genuinely hard tickets waited behind them. Those are two different projects, and conflating them is how you end up with an agent that's technically busy and operationally useless.

Before we wrote a line of code, we set one guardrail. Deflection would only count if the customer did not reopen or escalate within 72 hours. A ticket the agent 'closed' that came back angry isn't deflection at all. All you've done is buy a delay and hand the customer a worse mood on top of it. That definition mattered more than any model choice we made later.

Why Version One Over-Answered — the Helpful-by-Default Trap

The first build did what most RAG-backed support agents do: it retrieved from the entire knowledge base and answered whatever it could stitch together. In a demo this looks fantastic. In production it's a liability, and we saw exactly the failure modes we write about in why RAG demos work and production doesn't.

Two things went wrong at once. First, the knowledge base had three years of accreted docs piling up, deprecated features and old pricing still sitting in there, and the agent would happily explain a workflow that no longer existed because the chunk still scored well on similarity. Second — and this is the one that hurt — the model treated every question as answerable. Asked about a refund policy that varied by contract, it invented a plausible general answer. Asked to cancel an account, it gave step-by-step instructions instead of routing to a human.

The model had no concept of a question it *shouldn't* answer. Helpfulness was the only objective it optimised, and helpfulness with no scope is indistinguishable from confident fabrication. Our early eval runs showed the agent producing fluent, wrong answers with high self-reported confidence — a pattern we've since written up in why LLM confidence scores lie.

An agent that treats every question as answerable isn't helpful. It's confidently wrong at scale.

Fix One: Scope Retrieval to Versioned Docs

The first change was to stop retrieving from everything. We rebuilt the index so every document carried a version and a status — current, deprecated, or internal-only — and retrieval filtered to current-and-public before ranking anything. Deprecated docs stayed in the store for lookup but were fenced off from customer-facing answers.

This sounds like plumbing, and it is, but it collapsed a whole class of errors. The agent could no longer explain a feature that shipped and then got pulled, because those docs weren't in the candidate set anymore. When the product team shipped a change, the doc version bumped, and the agent's answers followed within the same release. Not a quarter later.

Here's the heuristic I'd hand any team building this: treat retrieval scope as a product decision. Someone has to sit down and decide what the agent is *allowed to see* before you go anywhere near tuning how well it searches.

• →Tag every doc with version and visibility status; filter before ranking, not after.
• →Keep deprecated content indexed for internal lookup but fenced from customer answers.
• →Bump doc versions in the same release as the product change so answers never lag the feature.
• →Treat 'what can this agent retrieve' as a scoping meeting, not a config default.

Fix Two: Build a Refusal Path, Not Just an Answer Path

The second change was giving the agent an explicit, first-class way to say no. Out-of-policy requests — refunds, contract changes, account deletion, anything touching money or legal terms — route to a human with the context already summarised, and the agent tells the customer plainly that a person will handle it. No guessing. No improvised policy.

We implemented this as a routing decision that runs *before* the answer generation, not as a disclaimer bolted onto the end. The classifier's job is narrow: is this ticket in the set of things we've explicitly decided the agent can resolve? If not, it never reaches the answering path. This is the workflow-versus-agent distinction in miniature — we took agency away from the model exactly where autonomy created risk.

A clean refusal turned out to *raise* trust, not lower it. Customers strongly prefer a fast, honest 'a specialist will pick this up' over a fluent answer that turns out to be wrong. The refusal path is also where escalation-rage lives or dies: the tickets that make people furious are almost never the ones the agent handled — they're the ones it should have handed off and didn't.

Fix Three: Evals Built From Real Ticket Transcripts

We built the eval set from the client's own history — roughly 400 real, anonymised tickets sampled across categories, including the messy ones: multi-question threads, angry openers, tickets where the customer was wrong about what they wanted. Synthetic evals would have told us the agent was excellent. Real transcripts told us where it broke.

Each case was labelled with the correct outcome — resolve, refuse-and-route, or ask-a-clarifying-question — plus the ideal answer where one existed. We scored not just answer accuracy but *routing* accuracy, because sending a refund question to the answer path is a more expensive failure than a slightly-off answer to a how-to. For us the eval harness was the actual deliverable, a point we make at length in why the eval harness is the deliverable, not the agent.

The eval set also gave us an honest deflection ceiling before we shipped. When we labelled 400 tickets, only about half were purely informational and unambiguous. The rest needed account context or judgment, the kind of thing a human still has to do. That's what set the upper bound on what we could safely automate — not the model, the tickets themselves — and it's why we didn't promise 70%.

The Ceiling, and the Tickets We Chose Not to Touch

We landed at ~40% deflection sustained, measured on the 72-hour-no-reopen definition. We could have pushed the number higher by loosening the refusal path — and every point we'd have added there would have come from tickets the agent shouldn't have answered. The deflection ceiling isn't a modelling problem you engineer away. It's a property of your ticket mix.

We deliberately left several categories to humans. Anything involving money — refunds, billing disputes, plan changes. Anything with an emotional charge in the first message, detected cheaply on the opener. Anything account-specific that required data the agent couldn't safely read and act on. Automating these would have moved the deflection number and destroyed the CSAT number, which is the trade that quietly ruins most support-agent deployments.

We're candid about what's still hard. Emotional detection on the opener is coarse — it catches the obvious cases and misses the sarcasm. Multi-question threads still confuse the router more than we'd like. And keeping the eval set honest is ongoing work, because ticket patterns drift as the product changes; golden sets rot within months if nobody refreshes them. We ship monitoring alongside the agent precisely so the team catches failure before customers do.

The result that mattered to the client wasn't the 40%. It was that first-response time on the *remaining* tickets dropped by more than half, because the team stopped drowning in the easy forty questions. Deflection freed capacity. The refusal path protected trust. Neither works without the other.

The deflection ceiling isn't a modelling problem you engineer away. It's a property of your ticket mix.