Which approach addresses which failure
| Dimension | Retrieval (RAG) | Fine-tuning |
|---|---|---|
| Problem it solves | The model does not know facts specific to your organisation | The model does not follow your format, tone, or task convention reliably |
| What changes | The context supplied at question time | The model weights, through further training on examples |
| Keeping it current | Reindex the changed documents; the next answer reflects them | Retrain on updated examples; the current model keeps its old behaviour until you do |
| Access control | Enforceable at retrieval, per identity, per request | Not enforceable: training data is absorbed into weights and cannot be unlearned per user |
| Traceability | Answers can cite the passages they were built from | No citation is possible; the output cannot be traced to a source |
| Upfront cost | Ingestion, chunking, indexing and an evaluation set | A labelled example set, training runs, and evaluation |
| Running cost | Index storage, retrieval calls, and larger prompts per question | Lower per-token prompt cost; hosting a custom model if it is not managed |
| Latency | Retrieval adds a step before generation | No retrieval step, so typically lower latency per call |
| Changing base model | Portable: the index and retrieval layer are reusable | Tied to the model family; a new base model usually means retraining |
| Typical failure | The right passage was never retrieved, so the answer was unsupported | The model is fluent in your format and confidently wrong about your facts |
Name the failure before choosing the technique
The decision is determined by the failure you are seeing, and the two failures are distinguishable with one test. Paste the correct answer into the prompt by hand. If the model then answers well, it lacked knowledge and retrieval is the fix. If it still answers in the wrong shape, ignores your conventions or produces an unusable format, it lacked behaviour, and that is what fine-tuning addresses.
This test is worth running literally instead of reasoning about, because teams routinely misdiagnose the direction. A model that answers a policy question incorrectly looks like a model that needs training on your policies. Usually it is a model that was never shown them, and it would have answered correctly with the paragraph in front of it.
The reverse misdiagnosis is less common and more expensive. A model that produces correct facts in a format your downstream system cannot parse does not need a better index. Every additional retrieved passage makes the prompt longer and the output no more compliant.
- Wrong facts, right shape, and correct when the source is pasted in: a retrieval problem
- Right facts, wrong shape, unchanged when the source is pasted in: a behaviour problem
- Wrong on both, and no source exists in the corpus at all: neither technique applies yet
Retrieval is the default for enterprise knowledge, and the reasons are structural
For questions about organisational facts, retrieval is usually the right answer, and not only because it is cheaper to start. Three of its properties are difficult to obtain any other way: content can be updated by reindexing instead of retraining, access can be enforced per identity at query time, and the answer can carry the sources it was built from.
The access control property is the decisive one in most enterprises, and it is often noticed late. Fine-tuning absorbs its training data into the weights, which means a model trained on documents some users may not see will surface that content to everyone who can call it — not as a retrievable document, but as a fluent assertion with no source attached. There is no per-user filter to apply after the fact, and no practical way to unlearn a specific document.
Traceability matters for the same reason it matters in reporting: an answer that cannot be checked cannot be relied on where being wrong is expensive. Retrieval makes citation possible. It does not make it automatic, and citation correctness is measured separately from whether the answer is grounded.
Currency is the least glamorous advantage and often the most practically valuable. A policy changed this morning is reflected in this afternoon's answers if the index is refreshed. With a fine-tuned model the old behaviour persists until the next training run, and nothing in the output indicates that it is stale.
What fine-tuning is genuinely good at
Fine-tuning earns its place where the requirement is consistent behaviour rather than current knowledge: a strict output structure, a domain vocabulary the base model handles awkwardly, a classification or extraction task with abundant labelled examples, or a shorter prompt for a high-volume task where per-token cost matters.
The high-volume case is worth stating in economic rather than capability terms. If a task runs millions of times and needs a long instruction preamble every call, training that instruction into the model can move cost meaningfully. That is a throughput optimisation, made after the task works, and it is a poor first move.
Extraction and classification are the tasks where fine-tuning most often outperforms prompting outright. Given a few thousand labelled examples of your own documents, a smaller fine-tuned model frequently matches a much larger prompted one at lower cost and latency. The constraint is the label set, which is real work and is usually the reason the approach is not taken.
- A strict output format that prompting achieves only intermittently
- Domain vocabulary or conventions the base model handles awkwardly
- High-volume extraction or classification with a real labelled example set
- A long, stable instruction preamble that is worth removing from every call
Most production systems use both, in a specific order
The combination is not a compromise, it is the normal shape of a mature system: retrieval supplies the facts, and a fine-tuned or well-prompted model handles the behaviour. The order matters, and it is retrieval first. Build retrieval, measure it, and only then consider whether the remaining errors are behavioural.
Doing it the other way round is a common and expensive mistake. Fine-tuning first produces a model that is fluent in your house style and still wrong about your facts, which is harder to detect than being obviously wrong. The confident tone that fine-tuning produces is precisely what makes the remaining factual errors slip past a reviewer.
There is also an evaluation argument for the order. Retrieval quality is measurable independently, before any generation happens. Establishing that baseline first means a later behavioural change can be attributed, rather than confounded with an index that was never good enough.
The costs each one puts on you, and when
Retrieval front-loads engineering and spreads running cost across every query: ingestion, chunking, embedding, index storage, a retrieval call and a larger prompt each time. Fine-tuning front-loads data work and training, then runs cheaply per call. The asymmetry that decides most cases is maintenance: an index is refreshed, a fine-tuned model is retrained.
Retraining cost is frequently underestimated because it is not a one-off. Every material change to your content, your conventions, or the base model itself reopens it. Teams that fine-tuned to avoid the cost of building retrieval often find they have exchanged a running cost they can see for a periodic cost they forget to plan for.
There is a portability cost too. A retrieval layer is largely independent of which model sits behind it, so a better or cheaper model is a configuration change. A fine-tuned model is tied to its base, and when that base is deprecated the work is repeated.
- Retrieval: higher per-query cost, refreshed rather than retrained, portable across models
- Fine-tuning: lower per-query cost, periodic retraining, tied to its base model
- Both: an evaluation set built from real questions, which neither approach can skip
What to do before either, and it is usually the actual answer
A significant share of enterprise AI problems presented as a RAG-versus-fine-tuning decision are neither. The content is duplicated, superseded, unclassified or simply absent; there is no evaluation set, so quality is anecdotal; or the task is unbounded, so no architecture would produce a reliable result.
Retrieval inherits every weakness of the corpus beneath it. Indexing three versions of the same policy without knowing which is current produces a system that answers differently depending on which one is retrieved, and that is a content governance problem instead of a retrieval-strategy problem.
The evaluation gap is the more common one. Without a held-out set of real questions with expected behaviour, neither approach can be judged, and each change becomes a matter of impression. Building that set first is what makes the RAG-versus-fine-tuning question answerable with evidence rather than settled by preference.
Keep the evidence fixed when comparing behavior.
A single successful paste-in test is a diagnostic clue, not a release decision. In this hypothetical policy-extraction task, the output must contain an applicable rule, its source and a fixed set of fields. The comparison separates the effect of supplied evidence from the effect of training, so a newer corpus or an easier test set cannot be mistaken for a fine-tuning improvement.
Scope and assumptions: This is an experiment design without invented scores, training runs or a preferred vendor. Use only content authorized for each intended purpose; permission to retrieve a document does not establish permission to train on it. Model eligibility, retention, costs and available training methods require a current platform-specific check before execution.
| Dimension | Comparison arm | Controlled evidence and interpretation |
|---|---|---|
| Shared task | Freeze the question set, output fields, expected source versions, reader scope and grading rubric. Record the base model and the training lineage of any tuned candidate. | Separate development, validation and final held-out cases. Group near-duplicate documents or conversations appropriately so a lightly reworded training example does not masquerade as an independent test. |
| Prompted baseline | Use the untuned model with the agreed task instructions and no retrieved corpus. Preserve a legitimate clarification or refusal when the required facts are absent. | Measure structural compliance separately from factual support. An unsupported but neatly formatted answer must not count as a successful baseline. |
| Retrieval only | Use the untuned model with the permitted evidence captured for each question. Keep a record of both the retrieved candidates and the passages actually supplied. | Compare support and applicability with the baseline. If the needed condition is missing from the supplied evidence, a training experiment cannot repair that retrieval record. |
| Tuning only | Use the tuned candidate on the same task without retrieved corpus evidence. Keep evaluation questions and answer keys outside its training examples. | Check whether format or task convention improves without rewarding memorized policy facts. This arm does not establish current knowledge, per-reader authorization or a traceable source for an answer. |
| Combined candidate | Supply the tuned and untuned retrieval arms with the same recorded passages, reader scope and output requirements. Record any necessary prompt or configuration difference as part of the comparison. | Evaluate the remaining behavior difference while holding evidence constant. Then test the real retrieval path separately; a controlled comparison of fixed context is not an end-to-end production evaluation. |
| Decision and holdout | Choose the smallest change justified by the named failure. Use the final held-out set after development decisions, and retain unsupported, ambiguous and denied cases. | If formatting improves while source support or refusal behavior worsens, keep that trade-off visible. Record the option rejected and why; a combined design is a candidate to test, not an automatic destination. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: the combined candidate produces the required fields, but copies a retired rule from its training examples instead of using the current permitted passage. Structural compliance passes and factual support fails. Do not report the result as a general accuracy improvement. Preserve the changed-policy fixture, compare the supplied context and training lineage, and decide whether any added tuning is justified.
- Keep the test corpus and its applicability rules versioned separately from training data. A source update needs its own regression cases even if no model weights changed.
- OpenAI's official examples distinguish specialized behavior from factual recall and describe training/validation data separation. Those examples do not establish today's model availability, a universal sample-size requirement or an expected performance gain for this task.
Carry the supported decision and its unresolved failure cases into the AI application acceptance requirements rather than treating a preferred technique as the acceptance criterion.
Official references for this example:
Questions and trade-offs
Question
Can fine-tuning teach a model our internal facts?
It can absorb them, and it is a poor way to hold facts that change. Knowledge trained into weights cannot be updated without retraining, cannot be filtered per user, and cannot be cited. Facts belong in a retrievable store; behaviour belongs in the model.
Question
Is RAG always cheaper?
No. Retrieval costs more per query, because every call carries a retrieval step and a longer prompt. It is usually cheaper overall for knowledge that changes, because the alternative is a retraining cycle each time the content moves. For a high-volume, stable task, fine-tuning can be materially cheaper per call.
Question
What about a very large context window instead of retrieval?
Large contexts reduce how much retrieval precision matters; they do not remove the need for it. Cost scales with tokens supplied, relevant passages can be buried among distractors, and there is still no per-identity access filter on a corpus you paste in wholesale. Treat a large context as relaxing the top-k trade-off, not as replacing the retrieval layer.
Question
How many examples does fine-tuning need?
It depends on the task, and the honest planning answer is more than the first estimate. Format and tone adaptation can respond to a few hundred consistent examples; extraction and classification usually want thousands. The quality and consistency of the labels matters more than the count, and inconsistent labels train inconsistency.
Question
Can we start with fine-tuning and add retrieval later?
You can, and the order costs you. A fine-tuned model is fluent in your conventions while still being wrong about your facts, which is harder to catch in review than an obviously wrong answer. It also confounds evaluation, because a retrieval baseline was never established to attribute later changes against.
Question
Which is better for a regulated environment?
Retrieval, in almost every case, and for reasons of evidence rather than accuracy. Access can be enforced at query time per identity, the corpus can be audited and corrected, content can be withdrawn, and answers can cite their sources. A fine-tuned model offers none of those properties, and cannot unlearn a document that should not have been included.
Question
How do we decide without building both?
Run the paste-in test on a sample of real failures. If supplying the correct source fixes the answer, the gap is knowledge and retrieval addresses it. If the answer is still unusable with the source present, the gap is behaviour. That test costs an afternoon and resolves most of these decisions.
