The Agent That Passed Every Eval and Rotted Anyway

Three Things Move Under You After Launch

A production agent is never a fixed artifact. At least three inputs to its behaviour keep moving after you ship, and each one degrades quality in a way that leaves no obvious trace.

The first is your own changes. Someone tightens a prompt to fix one edge case, and quietly regresses four others. This is the most common source we see, and the most survivable, because at least there's a commit to point at.

The second is the vendor. When you call a hosted model by an alias — the ones that don't pin a dated snapshot — the weights under that alias change without a changelog entry that means anything to you. Your code is byte-for-byte identical and the behaviour is different. The third is the world: the distribution of inputs shifts as your product grows, new customer segments arrive, and the questions people ask stop resembling your golden set. That last one is the same rot we've written about in how golden eval datasets silently decay — except here it compounds with the other two.

None of these three fire a deploy hook. That's the whole problem. Your CI runs on code changes, and two of the three biggest drivers of drift never touch your code.

Your code is byte-for-byte identical and the behaviour is different.

The Day a Vendor Update Cost Us 11 Points

We had a document-extraction agent in production, stable for weeks, calling a hosted model through a floating alias. One morning the frozen regression suite — the one we replay on a schedule, not just on deploys — came back red. Accuracy on the held-out set had dropped by 11 points overnight. No commit. No config change. Nothing on our side had moved.

The vendor had quietly rolled the model out under the alias. The new version scored 'better' on the benchmarks vendors care about, and it was worse on the specific structured-extraction task we needed — in ways that only turned up on our own slices. Because we had the previous model version pinned in our snapshot, we could diff the two runs on identical inputs and confirm the regression came from the model. Not us. We pinned back to the dated snapshot within the hour and re-qualified the new version on our own suite before adopting it deliberately.

Here's the uncomfortable part. Without the scheduled replay, we'd have found out from a downstream data-quality complaint weeks later, with no way to attribute cause. The suite is what turned an invisible eleven-point regression into a red check with a diff attached. That's the difference between a monitoring story and a firefighting story.

What to Snapshot: Prompt, Model Version, Frozen Suite

A regression gate is only as good as the thing it holds constant. If any input to your agent can move silently — the model, the retrieval index, a config someone tweaked at 2am — you need it captured in a snapshot that travels with the version. We freeze three things and version them together.

Snapshotting the eval suite is the part teams skip. A live eval set that you edit as you go can't detect regression, because you can't tell whether the score moved or the ruler did. The frozen suite is your ruler. You add to it deliberately, version the additions, and keep the historical baseline intact so scores stay comparable across months — the same discipline that makes the eval harness the real deliverable rather than the agent itself.

• →The prompt and full config — temperature, tool definitions, retrieval parameters, everything that shapes output — hashed and committed.
• →The exact model version, pinned to a dated snapshot, never a floating alias. If you must use an alias, log the resolved version on every run so you can attribute drift.
• →A frozen eval suite with inputs and expected outputs, versioned alongside the prompt, with historical results preserved so today's score is comparable to last month's.

Replay on Every Change — and on a Clock

Two triggers, because there are two kinds of drift. On every change to a prompt or the code, replay the frozen suite, and block the merge if it regresses past threshold. This is standard CI, just with an eval suite instead of unit tests. It catches the regressions you inflicted on yourself before they ship.

The second trigger is the one almost nobody has: replay on a schedule, against production, with nothing on your side changed. Nightly is a reasonable default; more often if you're on a floating model alias or a fast-moving input distribution. This is the only thing that catches vendor updates and distribution shift, because neither one produces a commit to hang a CI run on. The scheduled replay is your smoke detector for the two-thirds of drift that never touches your repo.

Pair the frozen suite with a live sample of real production traffic scored on the same rubric. The frozen suite tells you whether known-good behaviour still holds; the live sample tells you whether the questions have moved somewhere your suite doesn't cover. When the two disagree — frozen suite green, live sample sliding — that's your signal that the distribution has drifted and the golden set needs refreshing. We go deeper on the online side of this in catching agent failure in production before users do.

Where Alerting Thresholds Actually Belong

Absolute score thresholds are the wrong default. 'Alert if accuracy drops below 85%' misses the agent that slid from 96 to 87 — still above the line, but nine points into a trend that ends badly. What you want to alert on is relative drift from the last-known-good baseline: a drop of more than N points versus the last green run, regardless of the absolute number.

Put thresholds per slice, not just in aggregate. An eleven-point drop on one document type can hide inside a flat aggregate if that type is a small fraction of the suite. Most of the regressions worth catching are concentrated in one place — one category, one customer segment — and averaging washes them out. Slice your suite by the dimensions that matter to the business and gate each slice on its own.

One caution: don't wire your gate to the model's self-reported confidence. Those numbers are badly calibrated and will lie to you, which we've covered in why LLM confidence scores break your human gate. Gate on measured correctness against known-good outputs, not on the model's opinion of itself.

• →Alert on relative drop from last-known-good, not an absolute floor.
• →Gate per slice — document type, intent, segment — so concentrated regressions can't hide in the average.
• →Fail the merge on CI-triggered regressions; page a human on scheduled-run regressions, since those imply something outside your control moved.

The Honest Part: Most Teams Have Zero Coverage Here

Across our audits, the pattern is consistent: teams have thorough unit tests for their code and nothing at all for the behaviour of the model they're wrapping. They tested the agent once, at launch, and treated that as a property of the system rather than a snapshot of one moment. Every input that could drift is drifting unobserved.

What's still genuinely hard, even with a good gate: knowing what to put in the frozen suite so it represents behaviour you care about, and keeping it representative as the world changes without invalidating your historical baseline. A frozen suite that no longer resembles production traffic passes green while your users suffer. That tension between stability and freshness doesn't fully resolve. You manage it with a frozen core plus a rolling live sample, and you accept that some drift will always be caught late.

The move that pays for itself is the cheapest one: pin your model version, freeze a suite, and run it on a nightly clock against production. That single change would have surfaced our eleven-point vendor regression, and it surfaces the class of failure that otherwise reaches your customers before it reaches your dashboard.