Five RAG Failure Modes We Keep Finding in Audits

First, Instrument Retrieval Separately From Answers

The reason teams misdiagnose RAG is that they only look at the final answer. A bad answer has two possible sources — retrieval put the wrong thing in the context, or generation fumbled the right thing — and end-to-end eval collapses both into one number. You can't fix a system whose failure you can't localise.

Before diagnosing anything, we log the retrieved chunks for every failing query and ask one question: was the passage that contains the answer present in the context at all? That single check partitions the entire problem space. If the passage isn't there, no model upgrade will save you — the fix is upstream in chunking, embedding, filtering, or freshness. If it is there and the answer is still wrong, you have a synthesis problem. This is the same discipline we argue for in why RAG demos work and production doesn't: retrieval needs its own scoreboard.

Before you upgrade the model, prove the right chunk was even in the window.

• →Log retrieved chunk IDs and text for every eval query, not just the answer.
• →Score retrieval recall independently: was the gold passage in the top-k?
• →Only debug synthesis on queries where retrieval demonstrably succeeded.

Failure One: Chunking That Severs Context

Symptom the team noticed: answers that are confidently half-right. The model quotes a policy but misses the exception clause, or cites a number without the condition that qualifies it. It feels like the model "isn't reading carefully."

Root cause we diagnosed: fixed-size chunking — 512 tokens, hard split — that cut a logical unit in half. The exception lived in chunk N+1, the rule in chunk N, and only one of them got retrieved. A table's header row landed in a different chunk than its data. The context was severed at ingestion, long before the model ever saw it.

Cheapest fix: chunk on structure. Split on headings, list boundaries and table units, and keep a small overlap (10–15%) so a sentence that references the prior clause still carries it. For structured docs, prepend the section heading to every child chunk so a retrieved fragment knows where it lives. It's an ingestion-pipeline change measured in days. In my experience it moves retrieval recall more than any embedding upgrade.

Failure Two: Embedding Mismatch With Domain Vocabulary

Symptom: retrieval that works fine on generic questions and falls apart on the ones that matter. Ask about "vacation policy" and it nails it; ask using the internal term "PTO carryover cap" and it returns something plausible but wrong.

Root cause: a general-purpose embedding model that has never seen the client's vocabulary. Part numbers, internal acronyms, drug names, legal citations — these collapse into near-identical vectors because the model has no signal to separate them. In one audit I ran, three distinct product SKUs sat within a hair of each other in embedding space, close enough that the retriever treated them as synonyms and served the wrong one on nearly every query. Semantic search is only as good as the semantics the encoder actually learned.

Cheapest fix: usually not fine-tuning. Add a hybrid retriever — BM25 or a sparse lexical index alongside the dense one — so exact-match terms like SKUs and codes are caught by keyword search while the dense side handles paraphrase. Reciprocal rank fusion to combine them is a well-worn, cheap technique. If a domain is genuinely dense with jargon, then fine-tune the embeddings, but hybrid retrieval clears most of the problem for a fraction of the effort.

Failure Three: Missing Metadata Filters

Symptom: the system answers from the wrong document — the deprecated version, another region's policy, a different customer's contract. The content is on-topic and the citation looks legitimate, which makes this the most dangerous failure of the five.

Root cause: the index has no metadata, or it has some and never filters on it. Everything is a flat pile of vectors. A query about the 2024 US handbook can happily retrieve the 2021 EU one because they sit close together in the same space, and semantic similarity has no notion of "valid," "current," or "belongs to this tenant." In regulated or multi-tenant settings, that's not a quality bug any more. It's a compliance and data-isolation risk.

Cheapest fix: attach metadata at ingestion — version, effective date, region, document type, tenant — and enforce hard filters at query time before the vector search runs. Most vector stores support metadata pre-filtering natively. For tenant isolation this isn't optional; a retriever that can reach another customer's documents is a breach waiting to happen, which is why we treat retrieval scope the way you'd treat an AI agent like an untrusted insider.

Failure Four: Stale Indexes

Symptom: answers that were correct at launch and drift wrong over weeks. The team updated the source docs, the app still cites the old figures, and nobody can reproduce it on demand because it depends on what changed and when.

Root cause: the index is a snapshot with no reliable refresh path. Someone ran an ingestion script once at go-live. Since then documents changed in the source system, but nothing re-embedded them, and deleted documents were never removed from the index — so the retriever confidently serves content that no longer exists anywhere else. Retrieval drift is slow and silent, which is exactly what makes it dangerous.

Cheapest fix: make ingestion incremental and event-driven, or at minimum scheduled with deletion handling. Track a content hash and last-modified timestamp per document; re-embed on change, tombstone on delete. Add a freshness check to your eval suite — a few queries whose correct answer depends on recent updates — so staleness shows up as a failing test, not a support ticket. Golden sets rot the same way indexes do, which is why we version evals alongside the system and refresh them deliberately.

Failure Five: Synthesis That Ignores Retrieved Evidence

Symptom: the retrieval logs show the correct passage was right there in the context, and the model still answered from its parametric memory — sometimes contradicting the very text you handed it. This is the one case where the generation step is genuinely at fault, and it's the least common of the five.

Root cause: usually the prompt, not the model. The instruction is vague ("answer the question using the context") with no directive to abstain when the context doesn't support an answer, no requirement to cite, and often the relevant chunk is buried in the middle of a long context where models reliably lose the plot — the lost-in-the-middle effect is well documented. So the model falls back on what it already "knows," which is exactly when hallucinations wear their most confident face. Self-reported confidence won't rescue you here either, for the reasons we've written about calibration.

Cheapest fix first: tighten the prompt so it demands grounded, cited answers and abstains outright when there's no evidence. Cut top-k. A shorter context means less diluted signal, and putting the highest-ranked chunk first helps too. Then add a groundedness check — an eval or a lightweight secondary pass — that flags answers the retrieved text doesn't support. Only if grounded synthesis still fails after all that should you reach for a stronger model.

The Honest Part: Fixing Retrieval Is a Standing Cost

The five fixes above are cheap relative to a model swap or a re-platform — most are days of ingestion and prompt work. What's not cheap is keeping them fixed. Chunking strategy has to be re-tuned as document formats change. Embedding choices age. Metadata schemas drift as the business adds regions and product lines. Staleness never stops fighting you; you don't get to patch it once and walk away.

The unglamorous truth from our audits: RAG isn't a feature you ship, it's a retrieval pipeline you operate, with its own evals and its own on-call reality. The teams that stay out of trouble are the ones who built a retrieval eval harness early and can answer "was the right chunk retrieved?" on any query, any day. That instrumentation is the actual deliverable — the thing that lets you change chunking or swap an embedding model without flying blind.

So before you approve budget for a bigger model, run the one check: pull the failing queries, look at what got retrieved, and see whether the answer was ever in the window. In our experience four times out of five, it wasn't — and the fix costs a fraction of the upgrade you were about to buy.