When a Reranker Earns Its Latency (and When It Doesn't)

A Reranker Fixes Ordering, Not Recall

Here's the distinction that decides everything. A cross-encoder reranker takes a candidate set your retriever already returned and re-scores it by reading each query-passage pair jointly. It can only reorder what's already in the candidate list. If the right passage isn't in your top-50, no reranker on earth will surface it.

So the first diagnostic is trivial: pull your top-k for a set of real queries and check whether the answer-bearing passage is anywhere in the list. If recall@50 is bad, you have a retrieval problem, and a reranker is treating the wrong disease. If recall@50 is high but recall@3 is low, the right chunk is sitting in position 17 when you need it in position 1, that's the exact gap a reranker closes.

We treat this as a hard gate. Before anyone proposes a reranker, we build a retrieval-only eval that grades whether the right passage was even retrieved. It settles the argument in an afternoon instead of a sprint.

A reranker can only reorder what's already in the candidate list. If the answer isn't in your top-50, you're solving the wrong problem.

The Cheaper Fixes You Should Rule Out First

Two changes fix more retrieval problems than reranking does, and both are close to free at inference time.

The first is chunk size and boundaries. A dense retriever fails constantly because the answer got split across a chunk boundary, or the chunk is so large the relevant sentence is drowned out by surrounding text. We've watched precision jump from re-chunking alone, moving from fixed 1000-token windows to structure-aware chunks that respect headings and tables. No added latency, no per-query cost.

The second is hybrid search. Pure dense retrieval misses exact-match terms — product SKUs, error codes, policy numbers, names. BM25 catches those; dense catches semantic paraphrase. Fusing them (reciprocal rank fusion is a reasonable default) recovers a chunk of recall that people wrongly try to buy back with a reranker. In our experience, if you skip hybrid and jump straight to reranking, you're paying latency to reorder a candidate set that's already missing the answer.

• →Re-chunk first: test structure-aware boundaries and 2-3 chunk sizes against your retrieval eval.
• →Add hybrid (BM25 + dense) with rank fusion before any reranker, it fixes recall, which a reranker can't.
• →Only after both, measure whether recall@k is high and precision@3 is low. That gap is the reranker's job.

The Case Where the Reranker Actually Won

In one engagement, an internal knowledge assistant over a large, jargon-heavy corpus, we had the textbook reranker case. Hybrid search plus decent chunking got recall@20 to around 94%. The answer was almost always retrieved. But the answer-relevant passage landed in the top-3 only 61% of the time, and the generator was anchoring on whatever came first, so answers overreached against the wrong chunk.

Recall was solved. Ordering wasn't. We added a cross-encoder reranker over the top-20 candidates and answer-relevant-passage-in-top-3 went from 61% to 89%. Downstream faithfulness improved because the generator saw the right passage at the top instead of a plausible neighbour, the overreach failure where a RAG answer cites a real source and still lies dropped sharply.

The cost was real. The reranker added roughly 200ms at p50 and more at p95, plus a per-query inference cost. For an internal tool with a generous latency budget, that trade was obviously worth it. For a customer-facing autocomplete, it wouldn't have been.

The Decision Heuristic We Use

Three inputs decide it: corpus size, query ambiguity, and latency budget.

Corpus size and diversity drive how much reordering helps. A small, clean corpus where the top-3 is usually right doesn't need reranking, you'd be paying to reorder a list that's already correct. Large, semantically dense corpora with many near-duplicate passages are where a cross-encoder's joint scoring pulls its weight.

Query ambiguity matters because rerankers shine on underspecified or multi-intent queries, the ones where the bi-encoder's single vector can't separate close candidates. Precise, keyword-heavy queries are different — hybrid search alone often gets you there. Latency budget is the hard constraint. A 200-400ms reranker pass is fine for a research assistant and unacceptable for a typeahead. Measure your p95 budget before you commit, while you still can change your mind, rather than after users start complaining.

• →Reach for a reranker when: recall@k is already high, precision@3 is low, the corpus is large/dense, queries are ambiguous, and you have 200ms+ of latency headroom.
• →Skip it when: recall is the actual problem, the corpus is small and clean, queries are precise, or your latency budget is tight, fix chunking and hybrid search instead.
• →Always attach a number: report recall@k and precision@3 before and after. If precision@3 doesn't move meaningfully, rip the reranker out.

What's Still Hard

Rerankers don't fix a broken corpus. If your docs contradict each other or are out of date, the cross-encoder will happily rank a confidently-wrong passage first — you need conflict detection over the corpus itself, not a better ranker. And reranker quality drifts as your query distribution shifts; a model tuned on last quarter's queries can quietly degrade, which is why the reranker belongs inside the same regression gates as the rest of the pipeline.

There's also a tuning trap. Reranking the top-100 instead of top-20 can help recall-into-precision but multiplies cost and latency; the sweet spot is usually smaller than teams assume. Test candidate-set depth as its own parameter against your eval.

The honest next step is boring and it works: build the retrieval eval, rule out chunking and hybrid search, and only then measure whether a reranker moves precision@3 enough to justify its latency. Most of the time the cheaper fix wins. When it doesn't, the numbers make the case for you.