Topic 8.1
Why Models Need Your Data
In one line
A model knows only its training text, which ended months ago and never included your private documents. RAG finds the relevant pages at question time and puts them in the prompt, which is cheaper, fresher and easier to check than retraining.
Think of it like this
An open-book exam. A student who studied last year answers from memory and gets recent rules wrong. The same student with the rule book open on the desk looks up the page and answers correctly, and can point at the page as proof. RAG is the open-book version of a model. Your program is the helper who opens the book at the right page before the model starts writing.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Knowledge cutoff
- The date after which the model has seen no text. It cannot know what happened later.
- Retrieval
- Looking up the most relevant pieces of your own data for a question.
- RAG
- Retrieval-augmented generation: retrieve passages, put them in the prompt, let the model answer from them.
- Context window
- The most text (counted in tokens) a model can read in one call, prompt and answer together.
- Fine-tuning
- Continuing a model's training on your own examples. Good for style and skills, weak for facts.
- Grounding
- Basing an answer on supplied documents so it can be checked against them.
Step by step
01Step 1: ask the model about a shop it has never seen
The fake model below only knows what it "learned": a common return window. Asked about Nimbus Bikes, it answers from memory with no warning that it is guessing. A real model does the same thing with more polish.
02Step 2: paste the right document
Give the same model the shop's own sentence in the prompt. The answer changes to the document's 30 days. Nothing about the model changed: only its input did.
Answer using only this document.
Document:
Unused bikes can be returned within 30 days with the receipt for a full refund.
Question: How long do I have to return a bike?03Step 3: why you cannot always paste everything
Now scale up. A 400-page manual is 120,000 words, about 160,000 tokens. The example prints how little fits in an 8,000-token window and what pasting it all would cost per question at an example price. The answer is to paste only the relevant part, which means finding it first.
04Step 4: retrieval is the missing step
Between the question and the prompt sits a search step. It turns "how long to return a bike" into the right two or three passages. The rest of the phase builds that step with simple, inspectable code before any library.
05Step 5: know what stays hard
If retrieval returns the wrong page, even a perfect model answers wrongly (and confidently). If the answer is in no document, the model should say so instead of guessing. Those two problems are the rest of the phase.
Try it yourself
- 1
Change the document
Edit
SHOP_DOCUMENTso it says 45 days. Predict the "with the shop document" line, then run. What did you have to retrain? Nothing: that is the point of RAG. - 2
Shrink the window
Change
context_windowto 128000 and predict the percent that fits. Is pasting the whole manual now sensible? Look at the cost line before you answer. - 3
Compare languages
Run the example in Python, Java and JavaScript. Find the line in each language that formats the dollar amounts to four decimals.
Code & diagrams
A fake model that answers from the document when it is given one, otherwise from a frozen guess. The sizes use a rule of thumb (about 3 words per 4 tokens) and an example price.
Expected output
question: How long do I have to return a bike?
without documents -> From memory: 14 days
with the shop document -> From the document: Unused bikes can be returned within 30 days with the receipt for a full refund.
manual size: 120000 words, about 160000 tokens
context window: 8000 tokens, so 5.0 percent of the manual fits
paste everything: 0.0800 dollars per question
retrieve 3 chunks of 150 words (600 tokens): 0.0003 dollars per questionBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Retrieve the wrong document
In the first example, pass the shipping sentence as the context instead of the return sentence: set SHOP_DOCUMENT to "Express shipping takes 1 to 2 days and costs 12 dollars." and run it.
Myth vs fact
Myth
RAG makes the model learn my data.
Fact
Nothing is learned. The passages are in the prompt for one call, and the model forgets them afterwards. That is also why updates are instant: change the document, not the model.
Myth
Fine-tuning is the better way to teach it my documents.
Fact
Fine-tuning shapes behaviour and style. For facts that change and must be checked against a source, retrieval is cheaper, fresher and auditable.
Myth
A big enough context window makes RAG unnecessary.
Fact
You still pay for every token on every call, long prompts are slower, and models find a needle in a huge prompt less reliably. Retrieval keeps prompts small, cheap and focused.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cost and latency scale with prompt tokens. Retrieving 600 tokens instead of 160,000 is roughly a 260-fold cut in input cost per question (160,000 / 600), and a much smaller time to first token (Topic 5.1).
- ▸
RAG changes where facts live: in a document store you control, with access rules. Permission checks belong at retrieval time (only fetch what this user may see), because anything retrieved can end up in the answer.
- ▸
Treat retrieved text as untrusted input: a document can contain instructions aimed at the model (indirect prompt injection, Topic 10.2).
Remember this
- 1
Why this exists. A model's knowledge is fixed at training time. Two things are missing from it. First, anything newer than its knowledge cutoff (the date its training text ends): new prices, new policies, yesterday's news. Second, anything private: your help centre, contracts, tickets, code. When asked about those, it does not say "I don't know". It produces the most likely sounding answer, which is a guess (Topic 1.9 called this hallucination).
- 2
The idea in one line. Retrieval-augmented generation means: (1) retrieve the few passages most relevant to the question from your own documents, (2) augment the prompt by pasting them in, (3) generate the answer from them. The model is still the same frozen function. You changed its input, and the input is the only thing you control (Topic 2.1).
- 3
Why not paste everything? Every model has a context window, the maximum amount of text it can read in one call (Topic 1.5). A 400-page manual is about 160,000 tokens, far more than a small window, and even when it fits you pay for every token on every question and the model finds facts less reliably in a huge prompt. The example below does that arithmetic: retrieving three short chunks costs a few hundred tokens instead of a hundred thousand.
- 4
Why not retrain (fine-tune) the model on the documents? Fine-tuning (continuing training on your examples) is good at teaching a model a style, a format or a narrow skill. It is a poor way to teach facts: it is slow and costly, the facts blur into the weights where nobody can inspect them, you cannot show where an answer came from, and when a price changes you must train again. With RAG you edit a document and the next answer is right. Topic 13.5 compares the two properly.
- 5
What RAG does not fix. Retrieval can return the wrong pages, or nothing useful. The model can ignore the pages, mix them up, or answer beyond them. Two jobs follow: make retrieval good (Topics 8.2 to 8.4) and make the answer checkable and honest (Topics 8.5 and 8.6). Most RAG bugs are retrieval bugs, so you test retrieval on its own first.
- 6
The shape of this phase. All examples use one tiny made-up shop with eight short documents. Start with the old and strong idea, keyword search (8.2), then search by meaning (8.3), then cut long documents into pieces (8.4), assemble the pipeline (8.5), make answers cite their sources or refuse (8.6), see what a vector database adds (8.7), and finish with a minimal service (8.8).
Explain it without notes
Why can a model not answer questions about a company's private documents, and what does RAG change?
Why is retrieval usually better than fine-tuning for teaching a model facts?
If a model has a very large context window, why would you still retrieve instead of pasting everything?
Practice
A manual has 250 pages of 300 words. Using 3 words for every 4 tokens, how many tokens is it? What percent fits in a 16,000-token window? Compute in Python and print both (one decimal for the percent).
At an example price of 0.50 dollars per million input tokens, what does it cost to paste a 100,000-token manual into every one of 20,000 questions? Print the total to two decimals.
Trade-offs
- ↔
RAG adds a search step to maintain and test, but it keeps facts fresh, cheap and checkable.
- ↔
More retrieved text gives the model more evidence and more noise, and costs more per call. Retrieve enough, not everything.
- ↔
Fine-tuning can still help alongside RAG (for style or format), at the cost of training and re-training.
Done when you can
Done when you can explain the knowledge cutoff and why private data is invisible to a model.
Done when you can name the three steps of RAG: retrieve, augment, generate.
Done when you can compute the token size and cost of pasting a document set versus retrieving part of it.
Done when you can explain why fine-tuning is the wrong tool for changing facts.