RAG, fine-tuning or long context? A decision framework for enterprise architects

Three ways to make a model know your business. A comparison table, a six-question decision flow and the mistakes to avoid.

Almost every enterprise AI project reaches the same fork in the road: the model does not know our business. Our products, policies, contracts, tickets and terminology are not in its training data. There are three main ways to close that gap: retrieval-augmented generation (RAG), fine-tuning, and long context (simply putting the material into the prompt). Teams often pick one based on what they read last. This framework helps you pick based on the problem.

The three options in one paragraph each

RAG keeps your knowledge outside the model. At request time the application searches your content, selects the most relevant passages and adds them to the prompt. The model answers from that material and can cite it.

Fine-tuning changes the model itself by training it further on your examples. It is best at teaching behavior: a tone of voice, an output format, a classification scheme, a domain’s way of reasoning.

Long context skips retrieval and sends the whole relevant body of material (a contract, a codebase module, a set of policies) in the prompt. Modern models accept very large inputs, so this is often simpler than it used to be.

Side-by-side comparison

Question RAG Fine-tuning Long context
Best at Facts and knowledge that change Behavior, style, format, narrow skills Reasoning over a bounded set of documents
Freshness As fresh as your index Frozen at training time As fresh as the documents you send
Access control Enforce at retrieval, per user Hard: knowledge is baked into one model Enforce when choosing what to send
Citations and traceability Strong: answers point to sources Weak Good, if you ask for references
Cost profile Indexing pipeline + modest tokens per request Training runs + model lifecycle Many tokens on every request
Latency Retrieval step + generation Fastest at runtime (short prompts) Slower as inputs grow
Main effort Data pipelines, chunking, search quality Training data, evaluation, retraining Prompt design, cost control

A decision flow you can use in a design review

  1. Is the problem about knowledge or behavior? If the model gives wrong or missing facts, it is a knowledge problem: start with RAG or long context. If it knows enough but responds in the wrong format, tone or reasoning style, it is a behavior problem: start with better prompts and examples, and consider fine-tuning only if that is not enough.
  2. How large and how fast-changing is the knowledge? A few documents that fit comfortably in the prompt and change per request (this contract, this incident) suit long context. A large or growing corpus (all policies, all product documentation, years of tickets) suits RAG.
  3. Do different users have different access rights? If yes, you need per-user filtering before content reaches the model. RAG with permission-aware retrieval is the natural fit. Fine-tuning on mixed-permission data is a governance risk, because the model cannot “unlearn” what one group should not see.
  4. Do answers need to be auditable? Regulated domains usually need to show where an answer came from. Favor RAG or long context with citations.
  5. What are the cost and latency budgets? High-volume, low-latency use cases (classification, routing, extraction at scale) can justify fine-tuning a smaller model. Low-volume, high-value analysis can afford long context.
  6. Can the team operate it? RAG needs data engineering. Fine-tuning needs an ML lifecycle. Long context needs mostly prompt discipline. Choose what your organization can run reliably, not just build once.

The answer is often a combination

  • RAG + long context: retrieve generously, then let a large context window absorb more surrounding material so the model sees passages in their full setting.
  • RAG + fine-tuning: RAG supplies current facts; a fine-tuned model reliably produces your required format or domain-specific reasoning.
  • Long context first, RAG later: a sensible way to prototype. Prove value by sending documents directly, then introduce retrieval when volume, cost or access control demand it.

Common mistakes

  • Fine-tuning to teach facts. It is expensive, goes stale and cannot cite sources. Use it for behavior.
  • Treating RAG as “just add a vector database”. Most RAG quality problems are data problems: poor chunking, duplicate or outdated documents, missing metadata, and weak search. Combine vector search with keyword search and measure retrieval quality separately from answer quality.
  • Ignoring permissions until late. Retrofitting access control into a RAG index is painful. Carry document permissions into the index from day one and filter at query time.
  • Sending everything because the window is big. Large inputs cost more, slow responses and can bury the important passage. Bigger is not automatically better.
  • No evaluation set. Without a fixed set of test questions and expected answers, you cannot tell whether a change to chunking, prompts or models helped.

Architect’s checklist

  • Have we classified the gap as knowledge, behavior, or both?
  • Do we know the size, change rate and ownership of the source content?
  • Is access control enforced before content reaches the model?
  • Can every answer be traced to a source where the business needs it?
  • Do we have cost and latency targets per request, and have we tested them?
  • Is there an evaluation set, and does it run on every change?
  • Who owns the pipeline or model after go-live?

New to these terms? Start with A beginner’s map of generative AI: 12 concepts every architect should know. Building agents on top of your RAG system? Read Guardrails for AI agents next.