Topic 0.4
The Smallest RAG, Built by Hand
In one line
A complete RAG pipeline fits in forty lines: score passages by shared words, put the best one in a prompt, let the model answer and cite. It works, until a question uses different words from the document.
Think of it like this
Finding a recipe in a cookbook by flipping through and looking for the word you have in mind. Looking for "pancakes" works if the page says "pancakes". If the book calls them "crepes", you flip past the right page without noticing. The smallest RAG searches exactly like this.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Pipeline
- A fixed series of steps where each step's output is the next step's input.
- Tokenise (for search)
- Split text into comparable words, usually lower-cased, with punctuation removed.
- Word overlap
- A score: how many different words two texts have in common.
- Top k
- The k best-scoring passages. k = 1 means only the single best.
- Paraphrase
- The same meaning in different words: "forgot my login" and "reset your password".
- Silent failure
- A failure with no error: the program finishes and prints a wrong result.
- Stop words
- Very common words such as "the", "in", "get" that say little about a text's topic.
Step by step
01Step 1: turn text into comparable words
"Monthly" and "monthly" must count as the same word, and "cost?" must not differ from "cost". Lower-case the text and keep only runs of letters and digits.
import re
def tokenize(text):
return re.findall(r"[a-z0-9]+", text.lower())
print(tokenize("What does the monthly pass cost?"))
# ['what', 'does', 'the', 'monthly', 'pass', 'cost']02Step 2: score every passage
Turn both word lists into sets (so repeats count once) and count what they share. The example prints the score of all five passages for each question, so you can see *why* a passage was chosen.
def score(question, text):
return len(set(tokenize(question)) & set(tokenize(text)))03Step 3: keep the best and build the prompt
retrieve returns the highest-scoring passage. build_prompt wraps it with the instruction and the question. Keep these as two functions: later phases replace the first many times and barely touch the second.
04Step 4: a question that works
"What does the monthly pass cost?" scores 3 on passage D8 and at most 1 elsewhere. D8 is retrieved, and the answer is right and cited. Note that the question says cost and the passage says costs: those did not match. It worked anyway because other words did.
05Step 5: a question that fails silently
"I forgot my login, how do I get back in?" scores 2 on refunds (in, get) and 1 on password reset (forgot). The wrong passage is retrieved. The answer is about refunds, and it carries a citation, which makes it look trustworthy.
06Step 6: what would fix it
Ignoring small words would make D3 win this time, by one word. But ask "I can't sign in" and there is no shared word at all. The lasting fix is a score based on meaning. That is what an embedding model provides, and Topic 0.5 runs one on these same five passages.
Try it yourself
- 1
Reword until it works
Change the second question to
"How do I reset my password?"and run. Then try"I can't sign in". Which passages are retrieved? The user should not have to guess the document's wording. - 2
Ignore small words
Add a set
STOP = {"in", "get", "the", "a", "i", "do", "my", "how"}and remove those words insidescore. Does the login question now retrieve D3? By what margin? - 3
Print the prompt
Add a line that prints
build_prompt(question, doc)for the failing question. Read it as the model would. Given only that prompt, was the model's answer reasonable?
Code & diagrams
Look at the "shared words" line of the second question: the right passage (D3) loses to D1 by one small word.
Expected output
Question: What does the monthly pass cost?
shared words: D1=0 D2=0 D3=1 D4=0 D8=3
retrieved: D8
answer: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes. [D8]
Question: I forgot my login, how do I get back in?
shared words: D1=2 D2=0 D3=1 D4=0 D8=0
retrieved: D1
answer: A ride cancelled at least 2 hours before its start time is refunded in full. [D1]Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Count repeats instead of different words
In the Python tab, change score to count every matching word, repeats included: return sum(1 for word in tokenize(text) if word in set(tokenize(question))). Run.
Myth vs fact
Myth
If the answer has a citation, it is trustworthy.
Fact
A citation shows which passage the model read, not that the passage answers the question. Here the wrong passage is cited perfectly.
Myth
A bad answer means the model is bad.
Fact
Here the model did its job on the prompt it received. The retriever chose the wrong passage. Check retrieval first, always.
Myth
Word matching is useless, so skip straight to vectors.
Fact
Word matching is unbeatable for exact things: error codes, product names, part numbers. Good systems use both (Phase 6).
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the retrieved ids and scores for every question from day one. In this example one printed line (
shared words: D1=2 ... D3=1) explains the whole failure. Without it you would be staring at a wrong answer and blaming the model (Topic 11.6). - ▸
The failing question is a vocabulary mismatch: users describe problems in their words, documents describe solutions in the organisation's words. It is the central problem of search and the reason embeddings matter. Collect such pairs from real users: they become your evaluation set (Topic 9.2).
- ▸
Taking the maximum over all passages is a linear scan: fine for 5 passages, fine for 50,000, wasteful for 50 million. An inverted index (Topic 1.2) avoids reading passages that share no word with the question.
Remember this
- 1
Why build the smallest version. The fastest way to understand a system is to build the worst version that works, and then watch it fail. Every later phase of this course exists to fix a specific weakness of the program on this page. If you can point at the line that fails, you know what each fancier tool is for.
- 2
Step one of the pipeline: score. For each passage, count how many *different* words it shares with the question. This is word overlap. To compare words fairly, both texts are first lower-cased and split into words, a step called tokenising (Topic 1.1 does it properly). "What does the monthly pass cost?" shares
the,monthlyandpasswith the monthly-pass passage: a score of 3. - 3
Step two: retrieve. Take the passage with the highest score. Real systems take the best few, called the top k; here k is 1 to keep the failure easy to see.
- 4
Step three: build the prompt. The prompt has three parts in a fixed order: an instruction ("answer using only the source"), the source with its id, and the question. A function builds it, so the layout is the same every time and can be tested (LLM Engineering Topic 3.7).
- 5
Step four: generate. The fake model in the example does what a good real model does with this prompt: it answers from the source and adds the source id. It reads the prompt it is given and nothing else, which is why it will faithfully answer from the *wrong* source too.
- 6
The failure. Ask "I forgot my login, how do I get back in?". The right passage talks about resetting a password. It shares one word with the question,
forgot. The refunds passage shares two:inandget, small words that carry no meaning here. Refunds wins. The model then gives a fluent, cited, completely wrong answer. Nothing crashed. This is the most important failure in the course: retrieval failed silently, and generation made the failure look like a success. - 7
Two separate flaws. First, meaningless words (
in,get,the) count as much as meaningful ones. Phase 1 fixes that with proper keyword scoring. Second, and deeper, "login" and "password" are different strings with nearly the same meaning, and no amount of word counting can see that. Phases 2 and 3 fix that with embeddings, which the next topic lets you try for real.
Explain it without notes
Walk through the four steps of the smallest RAG pipeline.
Why did "I forgot my login, how do I get back in?" retrieve the refunds passage?
Why is this failure called silent, and why is that dangerous?
The model produced a wrong answer here. Is the model at fault? How would you prove it either way?
Practice
Write score(question, text, stop) that counts shared different words, ignoring any word in the set stop. Print the scores of the refunds and password passages for the login question with stop = {"in", "get", "i", "my", "how", "do"}.
Write retrieve_top(question, docs, k) returning the ids of the k highest-scoring passages, best first, ties broken by position in the list. Print the top 2 for both questions.
Trade-offs
- ↔
Word overlap: trivial to build and explain, exact on identical wording, blind to meaning.
- ↔
k = 1: smallest prompt and the easiest failures to see, but one retrieval mistake means a wrong answer. A larger k forgives mistakes and costs tokens.
- ↔
A hand-made stop-word list: a quick improvement, but it is guesswork and differs for every language and subject.
Done when you can
Done when you can write the score, retrieve, build-prompt and generate steps from memory.
Done when you can explain, with the two scores, why the login question retrieves the wrong passage.
Done when you can explain what a silent failure is and how logging scores exposes it.
Done when you can name the two flaws of word overlap and the phases that fix each.