The question we hear most: 'Should we fine-tune a model?' The answer, 90% of the time, is no — you should build better retrieval. Fine-tuning teaches style and format; RAG teaches facts. They solve different problems and only one of them goes stale as your data changes.
Choose RAG when: the knowledge changes monthly, sources must be cited, or you need domain answers today. Choose fine-tuning when: the task is a stable format (summaries, classification, structured extraction), latency or cost of a big model is a blocker, or you need a distinct tone the base model won't hold.
The pattern that wins for most products: a small, fine-tuned model for format and routing, backed by retrieval for facts, with a strong evaluator upstream. It's more engineering, but your costs drop 10x and your answers stay current.
Whatever you choose, the eval harness is non-negotiable. Models change, prompts drift, and 'it worked last month' is not a strategy. Golden datasets and regression evals are the only control you have.
Run the cost model before the architecture diagram. Fine-tuning means GPU hours per training run, plus storage and routing for multiple model variants, plus re-runs every time your corpus changes. Retrieval means an embedding pipeline, a vector store, and per-query token costs. At small scale the fine-tune feels cheaper; at real scale the retrieval bill grows linearly with value delivered, while the fine-tuning bill grows with your uncertainty.
Data requirements differ by an order of magnitude. A useful fine-tune wants thousands of high-quality input-output pairs, cleaned and consistent, which most organisations simply do not have lying around. A useful retrieval system wants a good corpus, sane chunking, and metadata. You can stand up credible RAG in a week on documents you already own.
Freshness decides most arguments. A fine-tuned model knows what it knew when you trained it, and every product change, policy update or price move silently rots its answers. Retrieval reads your current source of truth at query time. If your knowledge changes weekly, the discussion is already over.
The mature answer is usually both. Fine-tune the behaviour, retrieve the facts: a model tuned for your format, tone and tool-calling patterns, grounded by a corpus that supplies the numbers and the nuance. Teams that reach this stage report the hallucination rate falling while the answers stay current, which is the entire point.
Whichever path you choose, the evaluation harness is non-negotiable. Golden datasets, regression suites that run on every prompt change, and a human review loop on a sample of production traffic. Models drift, prompts get edited at 2am, and the version that worked last month is not a strategy.
Enjoyed this? Let's talk about applying it to your product.
