Topic 0.2
Why Models Need Retrieval
In one line
A model's knowledge stops on its training date, never included your private data, cannot be updated quickly and comes without sources. Retrieval closes each of these gaps at question time.
Think of it like this
A new employee who is brilliant but joined today. They read widely before joining, so they can talk about almost anything. But they have never seen your price list, they don't know the policy changed last month, and when a customer asks something they don't know, they are too eager to help and make up an answer. You would not fire them. You would hand them the handbook and say "look it up, and tell the customer which page".
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Knowledge cutoff
- The date a model's training data ends. It knows nothing newer.
- Private data
- Text that was never public, so no model was trained on it: your documents, tickets, wiki.
- Hallucination
- A fluent, confident statement from a model that is false or unsupported.
- Freshness
- How quickly a change in your documents shows up in answers.
- Citation
- A pointer from a statement in the answer to the passage it came from.
- Stale
- Out of date: true once, wrong now.
Step by step
01Step 1: a fact that changed
The monthly pass used to cost 20 dollars. It now costs 25. A model trained before the change has "20 dollars" in its weights and no way to know that is stale. It does not say "it was 20 when I last looked". It says "20 dollars".
02Step 2: a fact that was never public
Lumen Bikes' refund rule (full refund up to 2 hours before the ride) lives in an internal handbook. The model has seen thousands of *other* companies' refund policies, so it produces an average of them. The result sounds right and is wrong.
03Step 3: a question with no answer anywhere
"Do you rent scooters?" Nothing in the handbook mentions scooters. From memory, the model says yes, because bike-rental companies often do. With retrieval, the search comes back empty, and the program can say so honestly.
def answer_with_sources(question):
hit = retrieve(question)
if hit is None: # nothing found: refuse before calling the model
return "I can't find that in the handbook."
return first_sentence(hit[1]) + " [" + hit[0] + "]"04Step 4: the same three questions, with sources
Run the example. Each question is answered twice: once from the frozen memory, once from the handbook. The handbook answers are current, come from the private policy, carry a source id, and the scooter question gets an honest refusal.
05Step 5: updating knowledge is now editing a document
Change the price in passage D8 to 30 dollars and run again. The answer follows at once. No model was retrained. This is the practical heart of RAG: the knowledge lives in data you control.
Try it yourself
- 1
Change the price
In passage D8, change
25 dollarsto30 dollarsand run. Which of the two answers changed? What would it take to change the other one in a real model? - 2
Make scooters answerable
Add a passage
("D20", "We rent scooters at the central shop only."). Predict the new "with sources" line for the scooter question, then run. - 3
Find the refusal
Find the line that refuses. Notice that the decision is made by ordinary code, before any model would be called. Why is that cheaper and safer than asking the model to refuse?
Code & diagrams
The memory is a scripted stand-in for what a real model might say with no sources. The notes after the arrows name the gap each answer falls into.
Expected output
Question: What does the monthly pass cost?
from memory: 20 dollars. <- out of date: the price changed after training
with sources: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
Question: Is a cancelled ride refunded?
from memory: Yes, within 30 days. <- made up: the policy is private, the model never saw it
with sources: A ride cancelled at least 2 hours before its start time is refunded in full. [D1]
Question: Do you rent scooters?
from memory: Yes, scooters are available. <- made up: it sounds likely, so the model says it
with sources: I can't find that in the handbook.Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Remove the refusal
In the Python tab, delete the two lines in answer_with_sources that check if hit is None, then run.
Myth vs fact
Myth
Newer models have a later cutoff, so this problem is going away.
Fact
Every model has a cutoff, and your private data is never in any of them. A newer model is stale about last week and blind to your wiki, exactly like an older one.
Myth
If the model doesn't know, it will say so.
Fact
Sometimes. Models are trained to be helpful and fluent, and a plausible guess is often the most likely text. Design the refusal yourself.
Myth
Fine-tuning on our documents is the proper fix.
Fact
Fine-tuning is good at changing style and behaviour and poor at storing exact facts. It also can't cite, and every document change would need another round. Topic 0.3 compares them.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Stale answers are the most dangerous gap in production, because the old fact was once true and so passes casual review. Ask of every RAG design: when a document changes, how long until answers change, and how would we know they didn't (Topic 11.2)?
- ▸
A hosted model with built-in web search covers public freshness, not your private data, and you don't control what it retrieves. For anything you must be able to audit, own the retrieval.
- ▸
The refusal decision is best made in code from retrieval scores, before the model call: it costs nothing, can't be talked out of by a clever question, and is easy to test. The model's own "I don't know" is a second line of defence, not the first.
Remember this
- 1
Why this matters. Before building a retrieval system you should be able to say exactly which problem it solves, because RAG costs real engineering. There are four separate gaps, and it helps to name them, because each one shows up differently in a product.
- 2
Gap 1: the knowledge cutoff. Training ends on a date, called the knowledge cutoff. The model knows nothing after it. Worse, facts that changed after it are still stored in their old form: the model will state last year's price with full confidence. The first question in the example below shows this.
- 3
Gap 2: private data. Models are trained mostly on public text. Your handbook, tickets, contracts and wiki were never in it, and should not be. No amount of prompting brings out knowledge that was never put in.
- 4
Gap 3: made-up answers. When a model lacks a fact, it still produces the most likely-sounding continuation. This is called hallucination (LLM Engineering Topic 1.9). "Yes, within 30 days" is a very plausible refund policy. It is also invented. The danger is that it reads exactly like a true answer.
- 5
Gap 4: no sources. Even when a closed-book answer is right, you cannot tell *why* it is right. There is nothing to click, nothing to check, nothing to show an auditor. In support, law, medicine and finance, an answer without a source is often not usable.
- 6
What retrieval changes. With the right passage in the prompt, the model reads the current price instead of recalling the old one, reads your private policy instead of inventing one, and can point at the passage it used. When nothing is found, your code knows that *before* calling the model and can refuse. And updating knowledge becomes updating a document, which takes seconds, where retraining a model takes weeks and a great deal of money. LLM Engineering Topic 8.1 gives the short version of this argument; the rest of this phase turns it into something you can run.
- 7
What retrieval cannot change. If the answer is in no document, RAG has nothing to find. If your documents are wrong or contradict each other, the answer will be too (Topic 8.5). And retrieval only helps with *knowledge*. If the problem is that the model writes in the wrong tone or the wrong format, that is a prompting or fine-tuning problem (Topic 0.3).
Explain it without notes
Name four reasons a language model needs retrieval to answer questions about a company's policies.
Why is a stale answer more dangerous than an "I don't know"?
Why is editing a document a better way to update knowledge than retraining a model?
What kinds of problem does retrieval not solve?
Practice
Write gap(question, memory, docs) that returns "stale" if memory has an answer and the documents have a different one, "missing" if memory has no answer, and "same" otherwise. Represent both as dictionaries from question to answer.
Write answer_or_refuse(hit) that takes a (doc_id, text) tuple or None and returns the text followed by the id in square brackets, or the refusal sentence.
Trade-offs
- ↔
Answering from memory: fast and cheap, fine for general knowledge, unusable for private or changing facts.
- ↔
Answering from sources: current, private and checkable, but only as good as the search and the documents.
- ↔
Refusing more often: fewer wrong answers, more "I can't find that" for questions the documents do cover. The threshold is a product decision (Topic 8.4).
Done when you can
Done when you can name the four gaps: cutoff, private data, made-up answers, no sources.
Done when you can explain why a stale answer is worse than a refusal.
Done when you can say what changes in the system when a document is edited.
Done when you can name two problems retrieval does not solve.