Stop Fine-Tuning to Fix a Retrieval Problem

What Fine-Tuning Actually Does — and What It Can't

Fine-tuning shifts the model's behaviour: tone, format, the shape of its outputs, how it decomposes a task. It's good at teaching a model to always respond in your JSON schema, to adopt a house style, to stop hedging. It nudges the distribution the model samples from.

It does not reliably teach the model new facts. Cram your product catalogue into a fine-tune and you get a model that confidently paraphrases a snapshot of your catalogue from three weeks ago, with no way to tell you which parts are stale. Facts belong in a retrieval layer where you can update them without retraining and cite them at answer time.

We see the same misdiagnosis over and over. "The agent gives wrong answers about our pricing" is almost never a behaviour problem. The model behaves fine — it just never saw the right pricing document at inference time. That's retrieval. As we've written in our field guide to RAG failure modes, the answer being wrong and the answer never being retrieved look identical from the outside, and teams grade the wrong thing.

A fine-tune to fix stale answers gives you a model that confidently paraphrases last month's data with no way to flag what's out of date.

• →Behaviour, format, tone, task decomposition → fine-tuning is a candidate.
• →New or changing facts, freshness, citation → retrieval, every time.
• →One-off instruction the model already follows when told clearly → prompting.

The Four Costs Teams Forget to Price In

When someone scopes fine-tuning they price the training run. That's the cheap part. The expensive parts are structural and they don't go away.

Freshness is the first. A fine-tuned model is frozen at its training cut, so every time your underlying data shifts in any material way, you're back in the pipeline waiting on a re-training cycle. Retrieval updates in seconds.

Latency and serving come next. A custom fine-tuned weight often means you're off the fastest hosted endpoints and onto your own serving path, or a slower dedicated deployment. Teams that fine-tuned to "improve quality" sometimes ship a model that's both harder to host and no more accurate.

Then the pipeline itself. A fine-tune isn't a deliverable, it's a standing obligation: labelled data curation, a training and validation split you keep honest, a re-train every time the base vendor ships a new model you want to move to. And it needs its own eval harness to prove each new checkpoint didn't regress. You now own two moving systems instead of one.

Run the Eval That Should Gate the Decision

Before anyone writes a training script, we run two cheap experiments. They usually end the debate.

First, a retrieval-only eval. Ignore answer quality entirely and measure one thing: when the answer exists in your corpus, did the right passage make it into the context? We've covered why this comes before answer-quality grading — if retrieval recall is low, no amount of fine-tuning fixes it, because the model never sees the fact. This is the single most common root cause we find, and it's fixable in days with better chunking and metadata.

Second, a strong-prompt baseline. Take the best frontier model, write a genuinely good prompt with few-shot examples of the exact behaviour you want, and run it against your eval set. Most "we need fine-tuning for format compliance" problems dissolve here. If a careful prompt gets you to 95% on schema conformance, a fine-tune to reach 98% is rarely worth a permanent pipeline.

Only if both of those are maxed out and you still have a measurable gap does fine-tuning enter the conversation. And I mean measurable: show me a number on an eval set. A hunch that the model "feels off" doesn't count.

• →Retrieval recall below target → fix chunking, metadata, and ranking first.
• →Strong-prompt baseline closes the gap → ship the prompt, skip the fine-tune.
• →Gap persists after both, and it's a behaviour/format/latency gap → fine-tuning is now a legitimate candidate.

The Decision Checklist

Here's the checklist we actually walk through with teams. If you can't answer yes to the gating questions, you're not ready to fine-tune — you're ready to fix something cheaper.

• →Is the problem facts or freshness? If yes → retrieval, stop here.
• →Have you run a retrieval-only eval and confirmed recall is already high? If no → do that first.
• →Does a strong few-shot prompt on a frontier model close the gap? If yes → ship the prompt.
• →Is the remaining gap a behaviour, format, or latency problem you can measure on an eval set? Required yes.
• →Can you commit to re-training and re-evaluating on every base-model upgrade you want? Required yes — this is the standing cost.
• →Do you have enough clean, representative labelled examples (usually hundreds to low thousands of good ones, not ten)? Required yes.
• →Will the fine-tuned model be servable within your latency and cost budget? Required yes.

The Two Times Fine-Tuning Actually Won for Us

It does win. Twice in recent memory the evidence pointed there and the payoff was real. Both times the trouble was behaviour or latency, and knowledge had nothing to do with it.

The first was strict format and behaviour conformance. A classification-and-extraction agent had to emit a rigid structured output on every call, and even a carefully engineered prompt on a strong model left a stubborn tail of malformed outputs — around 4% failing schema validation, which downstream systems couldn't absorb. Fine-tuning a smaller model on a few thousand curated examples drove schema failures to well under 0.5% and held there across the eval set. Prompting had plateaued; the fine-tune cleared the tail.

The second was latency-driven distillation. A frontier model produced excellent results but at a p95 latency that didn't fit the interaction budget, and the per-call cost didn't scale. We used the big model to generate a high-quality dataset, then fine-tuned a much smaller open model to match its behaviour on that narrow task. The distilled model landed within a couple of eval points of the frontier model while cutting p95 latency by roughly 70% and dropping per-call cost by a similar margin. That's the classic distillation case, and it only worked because the task was narrow and the eval set was honest.

Notice what both have in common: a measured gap that prompting couldn't close, a narrow well-defined task, and no dependence on facts that change. That's the whole territory where fine-tuning earns its keep.

The Part That's Still Hard

The honest gap in all of this is data. A good fine-tune needs a clean, representative, correctly-labelled dataset, and assembling one is more work than most teams budget. Garbage examples produce a model that's confidently wrong in new ways, and you won't catch it unless your eval set is genuinely adversarial rather than a comfortable sample of the easy cases.

The other unsolved bit is upgrade churn. The base-model landscape moves fast. A fine-tune you shipped this quarter may be beaten by next quarter's frontier model with a good prompt — at which point your pipeline is a liability you're maintaining for no gain. Build in a periodic re-test against the current best off-the-shelf option, and be willing to retire the fine-tune when it loses.

The practical next step is boring and it works: run the retrieval-only eval and the strong-prompt baseline before you scope any training work. Most of the time you'll find your fine-tuning project was a retrieval bug wearing a bigger price tag.