Fine-Tuning, RAG, or Prompting, A Framework From Failure Modes
Pull-quote: “Choose the intervention by the failure mode, not by the fashion. Every one of these techniques fixes a specific kind of wrong.”
Why this matters
“Should we fine-tune?” is the most common first question from teams adopting LLMs, and it is almost always the wrong one. Fine-tuning, retrieval, and prompting are not competing philosophies; they are repairs for different defects. The right way to choose is to diagnose how the system is failing, then apply the intervention that fixes that failure, because each technique fixes exactly one kind of wrong and quietly makes some other dimension worse.
Diagnose first
Observed failure First intervention
──────────────── ──────────────────
Doesn't know the facts ──────► Retrieval (RAG)
(missing, stale, private
knowledge)
Prompting: examples,
Knows it, says it wrong ──────► output contract; light
(format, tone, style) tuning if prompts plateau
Reasons about it badly ──────► Better model, or decompose
(wrong approach, bad logic) the task into checkable steps
Right but too slow / costly ──────► Distill or fine-tune a
smaller model on the
validated behaviorThe mapping earns its keep in what it rules out. Fine-tuning does not reliably inject facts; a model tuned on your product docs still hallucinates the details it half-absorbed, while retrieval quotes them. Retrieval does not fix reasoning; handing a confused model more documents produces confusion with citations. And no prompt fixes a capability the base model lacks; past a point, prompt iteration on a reasoning failure is negotiating with the wrong model.
The honest cost table
Technique choice is a total-cost decision, and the costs that matter are the recurring ones:
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| Upfront cost | Hours to days | Weeks: corpus, chunking, index, eval | Weeks: data curation dominates |
| Recurring burden | Prompt regression suite | Corpus freshness pipeline, retrieval eval, index ops | Re-train on every base-model change; drift monitoring |
| Knowledge updates | Edit text, redeploy | Re-index the document | Full retraining cycle |
| Skill required | Every engineer | Data + search engineering | ML engineering + labeled data |
| Failure visibility | High: read the prompt | Medium: inspect retrieved set | Low: behavior baked into weights |
Two lines deserve emphasis. Re-train on every base-model change is the tax most teams discover late: the tuned checkpoint anchors you to a base model whose successors keep arriving, and each migration repeats the training and evaluation cycle. And failure visibility is why prompting and RAG dominate in regulated settings: you can audit a prompt and a retrieved set; you cannot audit a weight delta. It is also why governed deployments tend to route inference to a managed foundation-model endpoint inside their own platform rather than to custom-tuned weights. Governance prefers the interventions it can inspect.
The order of operations
In practice the framework collapses to a sequence. Prompt first, and take it seriously: a versioned prompt with good examples and a schema-enforced output contract resolves more production failures than teams expect. Add retrieval when the failures are factual. Reach for tuning last, when the behavior is validated and stable and the motive is economic: compressing a proven workflow into a smaller, faster model. Tuning as an aspiration, before the behavior exists, fails on data you do not yet have.
Closing
Fine-tuning, RAG, and prompting are answers to different questions: what should the model know, how should it behave, and what does that behavior cost to run. Diagnose the failure mode before choosing, price the recurring burden rather than the demo, and escalate in order of reversibility: prompt, then retrieve, then tune. The cheapest intervention that fixes the observed failure is the correct one, every time.