Define the Acceptance Test Before the Build, Not After

A Goal Is Not an Acceptance Test

'Reduce support load' can't be passed or failed. It just points you somewhere. You can ship something that technically reduces support load by 2% and deflects a handful of tickets, and both sides will still disagree about whether the project succeeded. That disagreement is baked in from day one because nobody agreed what the number was.

An acceptance test is different. It names the metric, the dataset, the threshold, and the person who signs off. 'The agent resolves ≥40% of a frozen set of 300 real, labelled inbound tickets end-to-end with no human escalation, at a false-resolution rate below 3%, judged by the head of support.' That sentence is boring, and that's the point. There's nothing left to argue about at delivery.

Turning the goal into that sentence is where the real work happens. You have to pick the metric that actually maps to business value — deflection, not 'answer quality' — and then decide what counts as a resolution and what counts as a failure that hurts. In our support agent work, the false-resolution rate mattered more than the deflection rate, because a confidently wrong answer that closes a ticket costs more than one that escalates. That's the kind of thing you only surface by forcing the test up front.

If nobody can write down the number that means 'this worked,' the project isn't ready to build.

• →Metric: the one number that maps to business value, not a proxy.
• →Dataset: a frozen, labelled set of real examples — not synthetic, not cherry-picked.
• →Threshold: the pass bar and the fail bar, separately.
• →Guardrail metric: the thing that must not get worse (false resolutions, latency, cost per task).
• →Sign-off: the named person who calls pass or fail.

The Golden Set Is the Forcing Function

The acceptance test is only as real as the data it runs on. So the first thing we ask for is 100-300 real examples of the task, labelled with the right answer. We want the actual tickets and documents and queries the system will hit in production, with ground truth attached. Slides and a happy-path demo don't count.

This request does more diagnostic work than any workshop. Sometimes the labelled examples don't exist, and nobody can produce them without weeks of effort, which tells you the organisation doesn't actually agree on what a correct outcome looks like. Sometimes you get the examples and half of them are ambiguous even to the human experts, which means an agent will never clear a clean threshold either. Either way, you've learned something that would have cost a quarter to learn during the build.

The golden set also has to survive contact with time. Golden eval sets rot within months as edge cases shift and the product changes, so we version it alongside the prompts and agree up front how it gets refreshed. The acceptance test isn't a one-time gate — it's the eval harness that becomes the actual deliverable, the thing that lets the team keep shipping after we're gone.

How an Up-Front Test Killed a Project

Here's an anonymised one. A mid-market company wanted an agent to auto-triage and respond to a category of inbound compliance queries — the pitch was 'cut the manual review queue in half.' Sounded like a clean automation win.

We asked for the golden set: 200 real queries with the correct disposition, labelled by their compliance team. Two things fell out of that exercise before a single line of agent code existed. First, when two compliance analysts labelled the same 50 queries independently, they disagreed on the correct disposition about 30% of the time. Second, a large share of the 'correct' answers depended on account context that lived in a system the agent would have had no clean way to reach — the classic place where agent projects actually stall.

The acceptance test we'd drafted — '≥70% agreement with the ground-truth disposition' — was unwinnable, because there was no stable ground truth to agree with. Human experts couldn't hit 70% agreement with each other. We didn't build the auto-triage agent. We reshaped the engagement into a much narrower assistive tool that surfaced the relevant policy passages and context to a human analyst, with a retrieval-quality bar instead of a disposition-accuracy bar. That version shipped and stuck. The full automation would have failed acceptance and everyone would have blamed the model.

Why the Fail State Matters as Much as the Win

A target with no defined failure is a wish. We insist on writing down what 'worse than doing nothing' looks like, because plenty of AI systems clear a headline metric while quietly making something else worse. Take a support agent that deflects 40% of tickets and torches CSAT because the 40% it deflected were the ones that most needed a human. Looks like a win on the dashboard. Isn't.

So every acceptance test carries a guardrail metric — the thing that isn't allowed to regress. False-resolution rate. Cost per resolved task. P95 latency. The set of actions the agent must never take on its own. Defining these before the build means you're not discovering at launch that your win came at an unacceptable price.

This is also where you decide whether the thing should be an agent at all. A lot of what arrives labelled as an 'agent project' clears its acceptance test more reliably as a constrained workflow with the agency taken out. Writing the fail state first tends to push you toward the narrower, more testable design — which is usually the one that ships.

The Checklist to Force This With Any Vendor or Team

You don't need us to run this. Any technical leader can impose the same rigour on an internal team or an external vendor before signing off on an AI build. If a vendor resists these questions, that's your answer about the vendor.

Run this before the work starts. By the demo it's too late.

• →Write the acceptance test as one sentence: metric, dataset, threshold, sign-off. If you can't, you're not ready to build.
• →Demand a frozen golden set of 100+ real, labelled examples. If it can't be produced, that's the first finding.
• →Have two experts label an overlap sample independently. If they disagree often, no model will hit a clean bar either.
• →Name the guardrail metric that must not regress — and its limit.
• →Define the fail state explicitly: what result means 'don't ship this.'
• →Set a week-one or two 'baseline eval' checkpoint — run the naive approach against the test before committing the full build.
• →Agree who signs off on pass/fail and on which dataset version, in writing, before contracts.
• →Decide up front how the golden set gets refreshed so the test survives past launch.

What This Doesn't Solve

The up-front acceptance test isn't a guarantee. A system can clear it on day one and rot over the following weeks as prompts get tweaked and vendor models update underneath you — which is why the test has to become a standing regression gate, not a launch-day ceremony. And a metric can be perfectly measured and still be the wrong metric; picking the number that truly maps to value is judgment, and you can get it wrong.

There's also a real cost to this discipline. Some clients want to skip straight to building because the test-writing feels like delay. It isn't delay — it's the cheapest week you'll spend on the project — but it does front-load conflict that would otherwise stay comfortably buried until it's expensive. Naming that trade honestly is part of the job.

The pattern holds anyway. Across our engagements, the projects that shipped and stuck were the ones where everyone agreed on the number before the work started. The ones that turned into disputes were the ones where 'success' stayed a feeling. Write the test first. If it can't be written, you've just saved yourself a quarter.