Your RAG Isn't Hallucinating — It Never Saw the Answer

Answer-Quality Evals Can't See the Real Failure

Answer-quality evals — LLM-as-judge, semantic similarity to a reference answer — grade the final string. Useful. But it collapses two completely different failures into one number, because when the answer is wrong, the eval can't tell you whether the model reasoned badly over good context or reasoned perfectly over context that never contained the answer in the first place.

Those two failures need opposite fixes. A reasoning failure is a prompt or model problem. A retrieval failure is an indexing, chunking, or ranking problem. If you can't distinguish them, you'll spend weeks tuning the generation half of the pipeline while the actual bug lives in the retrieval half. We've watched teams burn a full sprint on prompt engineering for a system whose real issue was a chunk size that split every answer across two fragments.

This is the same pattern behind why RAG demos work and production doesn't — the demo questions happen to retrieve cleanly, so nobody instruments the step that quietly breaks at scale.

A wrong answer is either a reasoning failure or a retrieval miss — and your answer-quality eval cannot tell you which.

What a Retrieval-Only Eval Actually Measures

A retrieval-only eval ignores the generated answer entirely. It asks one question: given this query, did the passage that contains the answer make it into the top-k chunks we handed the model? Three metrics cover most of what you need.

Recall@k is the headline number — of the questions where an answer-bearing passage exists in the corpus, what fraction had that passage inside the top-k retrieved. Set k to whatever your prompt actually stuffs into context (often 5 or 8). MRR (mean reciprocal rank) tells you where the good passage landed — recall@10 can look fine while everything useful sits at rank 9, buried under noise the model down-weights. And the blunt one we always compute first: answer-passage-in-context rate — a yes/no per question, no ranking, just "was it there at all."

These are cheap to compute and brutally clarifying. You run them without a single LLM call to the generator, which means they're fast and deterministic — you can rerun the whole suite after every chunking or embedding change in seconds.

• →Recall@k: fraction of questions where the answer-bearing chunk is in the top-k passed to the model.
• →MRR: how highly the answer-bearing chunk ranks — surfaces "present but buried" failures.
• →Answer-passage-in-context rate: the yes/no ceiling on how well generation can possibly do.
• →Run all three with zero generator calls, so they're fast enough to gate every retrieval change.

How We Label Ground-Truth Passages

The reason teams skip this is they assume it needs a labelling platform and a month. It doesn't. The workflow we run in audits is deliberately low-tech.

Start with 150–300 real questions. Pull them from your actual user logs if you have them — synthetic questions are too clean, and they hide the messy phrasing that breaks retrieval in the first place. For each question, a human who knows the domain finds the chunk (or chunks) in the corpus that actually answer it and records the chunk IDs. That's the whole ground truth: a mapping from each question to the set of chunk IDs that contain the answer.

Two rules make or break the labelling. First, label against your actual chunked corpus, not the source documents — you're testing the retrieval system that exists, chunk boundaries and all. Second, allow multiple valid chunks per question and mark when the answer is genuinely split across chunks, because a split answer is a chunking bug you want counted, not smoothed over. Once you have this, recall@k and MRR are a few lines of code. Keep the set versioned alongside your prompts — like any golden eval set, this one rots as your corpus and query mix drift.

The Numbers Where Recall and Accuracy Diverge — and Where They Don't

Here's the pattern we see across audits, with the specifics anonymised. On one internal knowledge-base agent, end-to-end answer accuracy was stuck at 62%. The team had already cycled through two models and four prompt revisions and got nowhere. So we built the retrieval eval, and the numbers told the story straight away: recall@5 was 61%, and the answer passage actually made it into context only 64% of the time.

Read those together. The model was correct on essentially every question where retrieval succeeded, and wrong on essentially every question where it failed. Answer accuracy wasn't capped by the model — it was capped by retrieval, almost to the point. No prompt was going to fix the ~38% of questions where the answer was never in the room.

We fixed retrieval: adjusted chunk boundaries so answers stopped splitting, added a re-ranker, tuned k. Recall@5 went from 61% to 89%. Answer accuracy moved to 86% with no change to the model or the prompt. The generation half had been fine the entire time. The gap between recall and accuracy tells its own story too — when recall is high but accuracy lags, then you have a genuine reasoning problem and the prompt/model work is finally worth doing. Most teams do that work first, in the dark, before knowing whether they even have that problem.

Fix Retrieval First — The Transferable Rule

The rule generalises past RAG: whatever your pipeline hands the model as context sets a hard ceiling on output quality, and you should measure that ceiling before you tune anything below it. Retrieval recall is that ceiling for RAG. Tool-call results are that ceiling for agents — which is why an agent that thrashes through 40 tool calls often has the same root disease: it can't reliably get the right information into context, so it keeps trying.

Operationally this means your eval ordering is fixed. Measure answer-passage-in-context first. If it's low, stop — do not open the prompt, do not benchmark a new model. Fix chunking, embeddings, and ranking until recall is where it needs to be. Only then does answer-quality work pay off, because only then is the residual error actually attributable to generation. This is the same discipline behind treating the eval harness as the real deliverable: the harness is what stops you optimising the wrong half of the system.

We're candid that retrieval evals don't solve everything. Building the ground-truth set is real human labour, and it goes stale as your corpus grows. Multi-hop questions — where the answer needs two chunks that each only make sense together — are genuinely hard to label and harder to measure with simple recall@k. And a high recall number says nothing about whether the retrieved passage is up to date; a stale-but-relevant chunk scores as a hit and still produces a wrong answer. Retrieval eval is necessary, not sufficient. But it's the cheapest, highest-leverage measurement most RAG teams are missing, and skipping it is why so many of them are tuning a model that was never the problem.

Whatever your pipeline puts in context sets a hard ceiling on output quality — measure that ceiling before you tune anything below it.