Your RAG Cites the Wrong Policy Because Your Docs Disagree

Retrieval Success Is Not Answer Correctness

Most teams grade RAG on whether the right chunk was retrieved. That's the right instinct — we've argued that retrieval eval has to come before answer-quality eval, because if the passage never showed up, nothing downstream matters. But there's a failure that sits above retrieval and below the model: the corpus itself is internally inconsistent.

When your knowledge base has a 2021 travel policy, a 2023 revision, and a Slack-exported FAQ that summarised the 2022 draft, all three are legitimate documents. All three chunk cleanly. All three retrieve on a query about mileage reimbursement. The embedding model doesn't know one is dead. It ranks by semantic similarity, and the outdated doc often wins because it was written in the exact language the user is now searching with.

Feed a language model three passages that contradict each other and it does what these systems always do: produces one fluent answer. It won't flag the conflict. It just picks, and that pick is basically a coin toss as far as correctness goes. What you've built is a system that takes a governance problem and launders it into a confident sentence.

The retrieval did its job perfectly and handed the model a lie.

Three Detection Strategies That Actually Earn Their Keep

You can't fix contradictory documents at query time, but you can detect them and refuse to pretend they don't exist. Across our engagements, three techniques do most of the work — and each has a sharp limit.

Conflict detection at index time is the one that pays off most. When you build the index, cluster chunks by topic and run a lightweight entailment or contradiction check across near-duplicates. If two chunks about the same entity assert different numbers or dates, flag the pair. You don't need a perfect NLI model. Even a targeted LLM pass over clustered candidates catches the egregious cases, and the output is a conflict registry you can act on before a user ever hits it.

Recency weighting and source authority scoring are the other two. Attach an authored/modified date and a source-tier to every chunk in metadata — canonical policy system beats wiki beats Slack export — and let both influence ranking. The catch: recency is only a heuristic. A freshly edited wiki page can be wrong and an old signed policy can be current. Weighting reduces the odds. It won't decide correctness for you. Use these three to narrow the blast radius. They won't resolve anything on their own.

• →Index-time conflict detection: cluster near-duplicate chunks, run a contradiction check, emit a conflict registry — fix before users hit it.
• →Recency weighting: date every chunk, decay stale sources in ranking — but never let recency alone stand in for correctness.
• →Source authority scoring: tier your sources (system-of-record > wiki > chat export) and encode the tier in metadata, not in a prompt.
• →Chunk provenance: carry the source doc, version, and effective date all the way to the citation so a human can adjudicate fast.

An Anonymised Case: Two Docs, One Refund Window

On a support agent build, two documents in the client's corpus disagreed on the return window for a product line. One said 30 days. A newer regional addendum said 14 for that market. Both retrieved on return questions. The agent had been answering 30 across the board, and nobody noticed until a spot-check.

The tempting fix is to pick a winner — prefer the newer doc, done. We didn't, because the newer doc was regional and the older one was still correct everywhere else. Silently choosing either one produces confident wrong answers in half the cases. Instead we surfaced the conflict: index-time clustering flagged the two chunks as contradictory on the same slot (return window), and we tagged both with their scope metadata.

At answer time, when both conflicting chunks made the context, the agent was instructed to name the conflict and its conditions rather than resolve it — "the standard window is 30 days; for [region] it's 14" — and, where scope was ambiguous, to escalate. That turned a silent error into a visible, correct, conditional answer. The deeper fix landed a week later when the client merged the addendum into the canonical policy with explicit regional clauses. The AI change bought time; the content change was the actual repair.

Surface the Conflict, Don't Resolve It in the Prompt

The strong temptation is to write a prompt that says "if sources conflict, prefer the most recent." Resist it. That pushes a governance decision into a stochastic system with badly calibrated confidence — the same reason LLM self-reported confidence lies and breaks human gates. The model will apply your rule inconsistently and you'll have no audit trail for why it chose what it chose.

Better to make the conflict a first-class signal in the pipeline. When retrieved chunks carry a conflict flag from the registry, branch: answer conditionally if scopes are clear, escalate to a human if they're not. That's the same uncertainty-and-blast-radius logic behind selective human-in-the-loop escalation — gate on the cases where being wrong is expensive, not on everything.

And instrument it. If your traces don't show which documents were in context and whether any were flagged as conflicting, a wrong answer takes a day to diagnose instead of minutes. We wire this into span-level tracing before go-live for exactly this reason: "why did it say 30 days" should be a two-minute lookup, not an archaeology dig.

This Is a Content-Governance Problem Wearing an AI Costume

Here's the part clients don't want to hear. The reason your corpus contradicts itself is that no human owns which document is canonical. Three teams published three versions, nobody deprecated the old ones, and the wiki is a graveyard of drafts that were never marked dead. No retrieval trick fixes that. You're papering over the absence of a document lifecycle.

We can detect conflicts, tier sources, and surface disagreements — that ships in weeks. But the durable fix is that someone gets accountable for the corpus: a canonical source of record, an effective-date convention, a deprecation process, and an owner who kills stale docs. That's unglamorous data-hygiene work, and it collides head-on with AI budgets that assumed the hard part was the model. It rarely is; the hard part is usually the data access and content layer underneath.

Be honest with your stakeholders about the split. The AI system can make conflicts visible and prevent confident wrong answers today. It cannot decide that the 2023 policy supersedes the 2021 one — that's a business call. Projects that treat corpus cleanup as out of scope ship an agent that faithfully reflects an inconsistent knowledge base, which is worse than no agent, because it launders the mess into authoritative-sounding prose.

No retrieval trick fixes a corpus that no human owns.

What's Still Hard

Conflict detection at index time is precision-recall work, and both dials cost you. Set the contradiction check loose and you drown reviewers in false conflicts — two chunks that look opposed but really just describe different scopes. Set it tight and you miss the subtle disagreements that matter most. There's no clean threshold. You tune it against a labelled set of real conflicts from your own corpus, and you accept that some slip through anyway.

Scope-aware conflicts are the genuinely unsolved edge. "30 days" vs "14 days" is easy to flag. "Applies to enterprise customers" vs "applies to all customers", where the second doc silently predates the enterprise tier — that needs domain knowledge no NLI model has. For those, the honest answer is to escalate and put a human who owns the corpus on it. A smarter model won't get you there.

The practical next step is small and immediate: run a contradiction pass over your existing index before you tune anything else. Count the conflicts. That number tells you whether you have a retrieval project or a governance project — and in our audits, it's a governance project more often than teams expect.