Topic 0.3
RAG, a Long Prompt or Fine-Tuning
In one line
There are three ways to give a model knowledge it lacks: paste all of it into the prompt, retrieve only the relevant parts (RAG), or fine-tune the model. Each fits a different situation, and the choice is mostly arithmetic.
Think of it like this
You need a friend's help with a contract question. If the contract is two pages, you hand over the whole thing. If it is a filing cabinet, you pull out the three relevant pages first. And if your friend keeps answering in the wrong style, no amount of paper helps: they need practice. Those are the three options: paste everything, retrieve, and fine-tune.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Context window
- The maximum number of tokens a model can take in one call, input and output together.
- Token
- The small piece of text a model reads and writes; roughly three-quarters of an English word.
- Corpus
- The whole collection of documents you want to answer from.
- Fine-tuning
- Continuing a model's training on your own examples, which changes its weights.
- Lost in the middle
- Models use facts at the start and end of a long prompt more reliably than facts in the middle.
- Input tokens
- The tokens you send. You pay for them on every call.
Step by step
01Step 1: measure the corpus in tokens
A page of plain text is very roughly 500 tokens. The Lumen Bikes handbook is about 60 pages, so about 30,000 tokens. A company's full documentation can be 100,000 pages: 50 million tokens. Estimate first; measure with the real tokenizer before deciding (LLM Engineering Topic 1.2).
02Step 2: does it fit?
Compare with the context window. With an example window of 128,000 tokens, the handbook fits four times over. A 2,000-page help centre (a million tokens) does not fit at all. For that corpus, pasting everything is not a choice.
03Step 3: what does each question cost?
Cost per question is input tokens times the input price. The whole handbook on every question is 30,200 tokens. Five retrieved chunks of 300 tokens plus instructions is 1,700 tokens: about one eighteenth of the cost, on every single question.
def cost(tokens_per_question):
return tokens_per_question / 1_000_000 * PRICE_IN_PER_MTOK * QUESTIONS04Step 4: what else changes
Latency grows with input size, so the pasted prompt is slower to start answering. Accuracy can drop as the prompt grows, because the relevant paragraph is one among hundreds. Against that, the pasted prompt can never *miss* the paragraph, while retrieval can.
05Step 5: choose, and write down why
For one policy document: paste it. For the handbook: either works today; RAG wins on cost and gives sources, and the handbook will grow. For the help centre and beyond: RAG is the only option. The playbook at the end of this phase turns this into a checklist.
Try it yourself
- 1
Use a bigger window
Set
CONTEXT_WINDOWto1_000_000. Which corpus now fits? Look at its cost per 10,000 questions. Fitting and being sensible are different questions. - 2
Retrieve more
Change RAG to ten chunks of 500 tokens. Predict the new cost before running. How does it compare with pasting the handbook?
- 3
Find your break-even
At what number of pages does pasting everything cost exactly twice what RAG costs? Work it out by hand, then check by adding a corpus of that size.
Code & diagrams
The window size and the price are example values, not any provider's real numbers. Put in your own before deciding anything.
Expected output
RAG sends 1700 tokens per question: 8.50 dollars per 10000 questions, whatever the corpus size.
one policy (4 pages): pasting everything sends 2200 tokens, fits, 11.00 dollars
the handbook (60 pages): pasting everything sends 30200 tokens, fits, 151.00 dollars
a help centre (2000 pages): pasting everything sends 1000200 tokens, does NOT fit, 5001.00 dollars
all company documents (100000 pages): pasting everything sends 50000200 tokens, does NOT fit, 250001.00 dollarsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the units
In the Python tab, change tokens_per_question / 1_000_000 to tokens_per_question / 1_000 (as if the price were per thousand tokens) and run.
Myth vs fact
Myth
Context windows are huge now, so RAG is obsolete.
Fact
A large window raises the size at which pasting is possible. It does not make it cheap, fast or accurate, and most company corpora are far larger than any window.
Myth
Fine-tuning a model on our documents makes it know them.
Fact
It nudges the model towards the style and topics of those documents. It does not reliably store exact facts, cannot cite, and goes stale with every edit.
Myth
RAG is always the professional choice.
Fact
For a small, stable document, pasting it is simpler, cannot miss, and is easier to debug. Choose RAG when size, cost, freshness or sources demand it.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Prompt caching changes the arithmetic for the paste-everything option: many providers bill a repeated prompt prefix at a lower rate (LLM Engineering Topic 6.6). A stable document pasted at the start of every prompt can be much cheaper than the plain sum suggests. Check your provider's rules before ruling it out.
- ▸
A common production shape is both: retrieval narrows millions of chunks to a few whole documents, and those documents go into a long prompt. Retrieval then only needs to find the right document, which is easier than finding the right paragraph.
- ▸
Fine-tuning does have a place in RAG systems: on the retriever side (an embedding model or re-ranker tuned to your domain's vocabulary) more often than on the generator. Phase 2 and Phase 6 come back to this.
Remember this
- 1
Why this choice exists. RAG is not free. You must prepare documents, run a search system and keep both up to date. Before building it, a good engineer asks whether something simpler works. Often it does, and saying so in a design review is a sign of experience, not laziness.
- 2
Option 1: paste everything (a long prompt). Modern models accept very large prompts, so a short document can simply go in whole, every time. Nothing to build, nothing to miss: the answer is definitely in the context. This is the right choice more often than people expect: one policy, one contract, one meeting transcript.
- 3
Where pasting everything breaks. Three limits. Size: the prompt must fit in the model's context window, the fixed budget of tokens for one call (a token is a small piece of text, roughly three-quarters of an English word; see LLM Engineering Topics 1.2 and 1.5). Cost and speed: you pay for every input token on every question, and longer prompts take longer to process. Attention: models find facts at the start and end of a long prompt more reliably than facts buried in the middle, an effect known as lost in the middle. More text is not more accuracy.
- 4
Option 2: retrieve (RAG). Send only the few passages that matter. The prompt stays small and roughly the same size whether you have sixty pages or six million. The price is the search system, and a new way to fail: the right passage may not be retrieved.
- 5
Option 3: fine-tune. Fine-tuning continues a model's training on your own examples. It is good at changing behaviour: tone, format, a classification task, a specialised vocabulary. It is poor at storing facts: the model blends what it learned, cannot quote a source, and every changed fact would need another training run. Fine-tuning and RAG are not rivals. They fix different problems.
- 6
A test you can apply. Ask four questions. Does it all fit comfortably in the prompt, at a cost you accept per question? Then paste it. Is the knowledge large, changing, or does it need sources? Then retrieve. Is the problem *how* the model answers and not *what it knows*? Then prompt better first, and fine-tune if that isn't enough. Do you need both knowledge and a special style? Combine them: a fine-tuned or well-prompted model reading retrieved passages.
- 7
The numbers decide. The example below does the sums for four corpus sizes with example prices. Notice two things: pasting stops being *possible* quite early, and it stops being *sensible* even earlier. Topic 12.8 returns to this trade-off with caching and very large context windows in the picture.
Explain it without notes
When is pasting the whole document into the prompt the right engineering choice?
Give three costs of a very long prompt.
Why is fine-tuning a poor way to teach a model your company's facts?
A team has 500 pages of documentation that change weekly and a model with a 200,000-token window. What do you recommend and why?
Practice
Write fits(pages, window, tokens_per_page=500, overhead=200) returning True when pasting the whole corpus fits the window. Test it for 60 and 2,000 pages with a 128,000-token window.
Write cost_per_question(tokens, price_per_mtok) and print, to five decimals, the cost of a 30,200-token prompt and a 1,700-token prompt at an example price of 0.50 dollars per million tokens.
Trade-offs
- ↔
Paste everything: nothing to build and nothing can be missed, but cost and latency grow with the corpus, and it stops fitting.
- ↔
RAG: small, steady prompts with sources, but you build and run search, and retrieval can miss.
- ↔
Fine-tuning: changes how the model behaves, but is slow to update, can't cite and stores facts unreliably.
- ↔
Combining them: best results for large products, at the price of more parts to operate and evaluate.
Done when you can
Done when you can estimate a corpus in tokens and compare it with a context window.
Done when you can work out the input cost per question for a pasted prompt and for RAG.
Done when you can explain "lost in the middle" and why more context is not always better.
Done when you can say what fine-tuning is good for and why it is not a store of facts.
Done when you can pick between pasting, RAG and fine-tuning for a described situation and justify it.