How We Cut an Agent's Token Bill 60% Without Losing Eval Points

The Setup: A Good Agent With a Bad Bill

The client ran a customer-facing support agent handling multi-turn resolution — read the ticket, pull relevant docs, call a couple of internal tools, draft a reply. It worked. CSAT was fine, resolution rate was fine. The problem was that per-conversation token cost had roughly doubled since launch even though the prompts hadn't changed, and finance had started asking uncomfortable questions about the trajectory.

This is the same shape we've written about before in an agent that 10x'd traffic and blew its budget: the pilot economics and the production economics are different animals. What looks like a fixed per-task cost is usually a variable one that grows with conversation length, retrieval hit rate, and tool chatter.

Median cost per resolved conversation was ~48K tokens. But the distribution had a nasty tail — the 95th percentile was north of 140K. Before touching anything, we instrumented the logs to attribute every token to a source: system prompt, conversation history, retrieved context, tool call payloads, model output. You cannot optimise what you haven't attributed. Most teams have never done this breakdown even once.

Where the Tokens Actually Went

The attribution was blunt and a little embarrassing, which is normal. The model's own output — the part that does the useful work — was under 8% of total spend. Everything else? Overhead the agent carried into the model on every single turn.

Here's the median breakdown we found across a sample of 2,000 production conversations:

• →System prompt: ~6,200 tokens, re-sent on every single turn. A four-turn conversation paid for it four times.
• →Conversation history re-injection: ~14K tokens median, because the whole transcript was replayed verbatim each turn with no summarisation or windowing.
• →Retrieval payload: ~18K tokens. The retriever returned the top 12 chunks, raw, un-reranked, often with heavy overlap between chunks.
• →Tool call overhead: ~6K tokens across an average of 4.1 calls, several of which returned enormous JSON blobs the agent used two fields from.
• →Model output: ~3.5K tokens — the actual answer.

The System Prompt Was a Document, Not an Instruction

The 6,200-token system prompt had accreted over months. Every time the agent misbehaved, someone appended a paragraph. It contained three worked examples, a full copy of the brand tone guide, and a list of edge cases most conversations never hit. All of it re-billed on every turn.

We split it. The stable core — role, format, hard constraints — stayed in the system prompt and shrank to ~1,400 tokens. The situational material (tone examples, edge-case handling) moved into retrieval, fetched only when the ticket classifier flagged it as relevant. Roughly 70% of conversations never needed it, so 70% stopped paying for it.

The lesson we keep relearning: a system prompt is not documentation. If a rule fires on 5% of traffic, it doesn't belong in the block you send 100% of the time. Move it behind a condition.

Retrieval Was the Single Biggest Win

The retriever was tuned for recall in the demo — grab twelve chunks and let the model sort it out. In production that meant paying to stuff mostly-irrelevant context into the window on every retrieval-heavy turn. Worse, the chunks overlapped. So the model kept seeing the same paragraph three times over, and we kept paying for the privilege.

We added a reranker and cut to the top 4 chunks after dedup. Input tokens from retrieval dropped by about a third. This is the failure mode we described in why your RAG demo works and production doesn't — recall-maximising retrieval is a demo habit that becomes a cost centre at scale.

Critically, accuracy did not drop. We'd assumed fewer chunks might hurt answer quality; the eval said otherwise, because the reranker was putting the genuinely relevant chunk in the window more reliably than dumping twelve raw ones ever did. Fewer, better beat more, noisier — and it was cheaper.

The model's own output — the part doing the useful work — was under 8% of total spend. The other 92% was overhead we re-billed every turn.

History and Tool Chatter

For conversation history, we stopped replaying full transcripts. After the third turn, older turns get compressed into a running summary the agent maintains, and only the last two turns stay verbatim. That capped history growth. Before, it climbed linearly with conversation length, which is what was fattening the long tail.

On tools, two of the four calls returned full API responses when the agent used a handful of fields. We added a projection layer that trims tool responses to the fields the agent's schema actually references before they hit the context window. One tool that returned a 4K-token order object now returns ~300 tokens. We also merged two sequential calls that were almost always made together into a single composite call, shaving a full model round-trip off most conversations.

None of this is clever. It's plumbing. But plumbing is where the money is — the exotic model-swap optimisations everyone reaches for first usually move the bill less than trimming what you send.

How We Proved Quality Didn't Regress

This is the part that separates a real optimisation from a hopeful one. Every change was gated behind an eval suite before it shipped. We had a frozen golden set of ~300 real conversations with graded expected outcomes, plus an LLM-judge rubric for tone and completeness, validated against human ratings so we trusted the judge.

The discipline was simple: run the baseline, apply one change, re-run the identical set, compare. If the score moved outside the noise band we'd established from repeated baseline runs, the change was rejected or reworked. We never bundled optimisations — each one had to earn its place independently, because bundling hides which change caused a regression. This is the eval harness as the actual deliverable, not an afterthought.

The final numbers: median cost per conversation fell from ~48K to ~19K tokens, a ~60% reduction, with the 95th percentile down harder still because we'd killed the history-growth and retrieval-bloat tails. The composite eval score moved from 0.87 to 0.88 — statistically indistinguishable, and if anything nudged up because the reranker sharpened retrieval. We kept the golden set honest by refreshing it against new production edge cases, since eval sets rot within months if you don't.

The Cost Audit You Can Run This Week

You don't need us to do the first pass. The attribution step is the one most teams skip and the one that surfaces 80% of the waste. Here's the checklist we run, in order:

• →Attribute every token by source across a real sample — system prompt, history, retrieval, tool payloads, output. Do this before touching anything.
• →Find what you re-send every turn. Any static block billed on turn N that was already billed on turn 1 is a candidate to move or cache.
• →Audit the system prompt for conditional rules. Anything that fires on a minority of traffic should move behind a classifier or into retrieval.
• →Check retrieval breadth vs. relevance. If you're returning top-K raw with no reranking, you're almost certainly overpaying — measure what accuracy actually needs.
• →Inspect tool response sizes against fields used. Project down to what the schema references; merge calls that always co-occur.
• →Cap history growth. Summarise older turns instead of replaying full transcripts on long conversations.
• →Gate every change behind a frozen eval set with a defined noise band. One change at a time, scored before and after.

What's Still Hard

The honest caveat: this worked because the client had reasonable logging and we could build a trustworthy eval set from real conversations. If your traffic is too varied to sample representatively, or your judge rubric isn't validated against humans, the measurement discipline gets shakier and so does your confidence that quality held. Cost optimisation without a trustworthy eval is just guessing with a lower bill.

The other open problem is drift. A 60% cut on today's traffic distribution doesn't stay a 60% cut. New ticket types, a changed retrieval index — all of it moves the numbers. So the attribution and eval discipline isn't something you finish and file away; it's a monitor you leave running, because the distribution keeps shifting under you whether or not you're watching. We wired token-attribution dashboards into the client's observability so the next creep gets caught in weeks, not quarters.

If your agent works but the bill is climbing faster than usage, start with attribution. You'll almost certainly find the same four culprits — and most of the fix is plumbing you can do without a model swap.