Topic 16.9
Retrieval-Augmented Generation from Scratch
In one line
RAG answers questions from your own documents: split them into chunks, embed them, retrieve the chunks closest to the question, put them in the prompt as numbered sources, and have the model answer and cite them. It is a few dozen lines of Python, and most failures come from bad chunks and missing context.
Think of it like this
An open-book exam. The student does not memorise the whole library. They look up the right pages, read them, then write an answer and note which page it came from. Retrieval-augmented generation (RAG) is the open-book exam for a language model: look up first, then answer from what was found.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- RAG
- Retrieval-augmented generation: look up relevant text first, then ask the model to answer from it.
- Chunk
- A piece of a document, small enough to embed and put in a prompt.
- Overlap
- Words repeated between neighbouring chunks so facts at a boundary are not lost.
- Retrieval
- Finding the stored chunks most similar to the question.
- Grounded answer
- An answer based on the supplied sources instead of the model's memory.
- Citation
- A pointer from a statement in the answer to the source it came from.
- Hallucination
- A fluent answer that is made up or not supported by any source.
Step by step
01The pipeline
Indexing happens once. The answering path runs for each question. Keep them as separate functions so you can test each.
02Step 1: chunk with overlap
The first example cuts a short text into chunks of 8 words that share 3 words with the previous chunk. The loop steps forward by size minus overlap and stops when a chunk reaches the end. Overlap must be smaller than size, or the loop never moves.
03Step 2: embed, retrieve, prompt, answer
The second example is the whole pipeline in about forty lines, with a fake model because a real one needs a key and the internet. It has a small corpus (the collection of documents) and a toy embedding built from the corpus vocabulary, with stop words (very common words such as "the" and "is" that carry little meaning) removed. Sentences are split with a regular expression, a text pattern (Topic 3.6). It also has a retrieval function with a minimum score, a prompt builder that numbers sources, and a scripted fake model that answers from source [1] and cites it.
The second question has no relevant document. Retrieval returns nothing above the threshold, the prompt has no sources, and the fake model says it could not find the answer. With a real model, the instruction "say so if the sources do not contain the answer" does the same job, and the threshold stops you from sending irrelevant text at all.
04Step 3: see how bad chunks lose the answer
The third example splits one sentence so that "port" and "5432" land in different chunks. With no overlap no chunk holds both, so no retrieval can ever return the full fact. Overlap fixes it. Test your chunker with questions whose answers sit near boundaries.
05Real SDK shape for the last step
Once the prompt is built, the model call is the one from Topic 16.3. Keep the retrieval and prompt-building functions free of any SDK so you can test them without a model.
hits = retrieve(question) # your retrieval
prompt = build_prompt(question, hits) # numbered sources + instructions
reply = client.chat.completions.create(
model="your-model-name",
messages=[{"role": "user", "content": prompt}],
)
answer = reply.choices[0].message.content
assert valid_citations(answer, len(hits)) # reject answers that cite sources you never sent06Evaluate retrieval separately
Make a small list of questions with the chunk that should be found. Check the "hit rate": how often the right chunk is in the top k. If retrieval is bad, no prompt wording will rescue the answer. If retrieval is good and answers are bad, look at the prompt and the model.
Try it yourself
- 1
Add a document
Add a fourth document to
DOCSabout a topic of your choice, then ask a question about it. Does retrieval find it? Does a question that only shares a stop word with it still match? - 2
Raise the threshold
Change
min_scoreto 0.6 and run again. Which questions now get "could not find"? What is the trade-off?
Code & diagrams
Expected output
1 | Docker images are built from layers. Each layer
2 | layers. Each layer is read-only. A container adds
3 | A container adds a thin writable layer on
4 | writable layer on top.Ask your own questions by changing the list at the bottom. Notice the "not found" answer when nothing relevant is retrieved.
Expected output
Q: what does a virtual environment do
retrieved: [('python.md', 0.53)]
A: A virtual environment keeps the packages of one project separate from another. [1]
Q: what is the capital of France
retrieved: []
A: I could not find that in the documents.Expected output
size=5 overlap=0
The default port for PostgreSQL
is 5432 and it rarely
changes
one chunk holds both 'port' and '5432': False
size=5 overlap=3
The default port for PostgreSQL
port for PostgreSQL is 5432
PostgreSQL is 5432 and it
5432 and it rarely changes
one chunk holds both 'port' and '5432': TrueBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Retrieve from an empty index
Search before anything has been indexed and use the first result without checking.
def retrieve(question, chunks, k=2):
words = set(question.lower().split())
ranked = sorted(chunks, key=lambda c: -len(words & set(c.lower().split())))
return ranked[:k]
sources = retrieve("what is the port", []) # nothing has been indexed yet
print("best source:", sources[0])Myth vs fact
Myth
RAG stops hallucinations.
Fact
It reduces them by giving the model facts, but the model can still ignore the sources, misread them, or answer when nothing relevant was found. Check citations and keep the "say so if absent" rule.
Myth
Bigger chunks are always better because they have more context.
Fact
Big chunks dilute the match, cost more tokens, and mix topics. Tune size on your own questions.
Myth
Retrieval quality is only about the embedding model.
Fact
Chunking, metadata filters, query wording, ranking and fresh data matter as much.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Common upgrades: hybrid search (keyword plus vector), a re-ranker that rescores the top 20 down to the best 5, query rewriting, metadata filters (tenant, date, language) and storing the parent document so you can show more context around a chunk.
- ▸
Permissions belong in retrieval: filter by what the asker may see before ranking, not after generation. Logging which chunks were retrieved for each answer makes bugs and leaks traceable.
- ▸
Keep the index and the documents in sync. Store a document hash, and re-embed only changed files. Record the embedding model name with the index, because a model change means re-embedding everything.
- ▸
Measure retrieval with numbers you can compare across changes. For a set of questions with a known right chunk, recall@k is the share of questions whose right chunk is in the top k, and MRR (mean reciprocal rank) is the average of 1 divided by the rank of the right chunk (0 if absent). Answer quality is a second, separate measure: is the answer supported by the supplied sources (faithfulness), and does it actually answer the question? Without the split you cannot tell a retrieval bug from a prompt bug.
- ▸
A citation proves only that the model pointed at a supplied source, not that the source supports the claim. Your code can check that the number exists and (cheaply) that quoted text really appears in that chunk. Checking that the claim is supported needs an eval with a judge or human spot checks.
Remember this
- 1
Why not just ask the model? A model knows only what was in its training data (the text it learned from), which is old and does not contain your private documents, and it may make up (hallucinate) a confident answer. And you cannot paste a whole company wiki into the prompt: the context window (Topic 16.4) is limited and every token costs money. RAG sends only the few relevant passages.
- 2
The pipeline has two phases. Indexing (done ahead of time): split each document into chunks, embed each chunk (Topic 16.8), and store the vectors with the chunk text and its source name. Answering (per question): embed the question, retrieve the top-k most similar chunks, build a prompt that lists them as numbered sources, call the model, and return its answer with the citations.
- 3
Chunking decides quality. Too big, and a chunk mixes topics and wastes the window. Too small, and the answer is cut in half, so no chunk contains it. Common practice is chunks of a few hundred tokens with a little overlap so sentences at the edges appear in two chunks. Splitting at natural boundaries (headings, paragraphs, sentences) beats cutting at a fixed count.
- 4
The prompt has fixed instructions ("answer using only the sources, cite their numbers, say so if the answer is not there"), the numbered sources, and the question. Numbering the sources gives the model something short to cite and lets your code check the citations: every number must point at a source you actually supplied (the practice shows this).
- 5
Retrieved text is untrusted data (Topic 16.7). A retrieved page can contain instructions aimed at the model. Mark sources as data, give the model no dangerous tools in a RAG answer path, and validate the output. Also respect access rights: retrieve only documents the current user is allowed to read, or RAG becomes a way to leak them.
- 6
What goes wrong: the relevant chunk was never retrieved (poor embedding or chunking); it was retrieved but ranked too low; the question needs information spread across several chunks; the documents are out of date; or nothing relevant exists and the model answers anyway. Defend with a minimum score so "no relevant source" produces "I could not find that", and measure retrieval separately from answer quality (Topic 16.11). A full Hectal course on RAG and agents is planned.
Explain it without notes
Walk through the indexing and answering phases of RAG.
How do chunk size and overlap affect retrieval quality?
How do you reduce wrong or ungrounded answers in a RAG system?
Users say your RAG assistant gives wrong answers. How do you find out whether the retriever or the generator is at fault, and what do you try first?
Practice
Write chunk_words(text, size, overlap) returning overlapping chunks of words.
Write build_prompt(question, sources) that numbers sources from 1.
Write valid_citations(answer, source_count) that checks that every [n] in the answer points at a real source and that at least one citation exists.
Trade-offs
- ↔
Few large sources or many small ones. More sources raise the chance the answer is present but cost tokens and add noise that the model may follow. Start with 3 to 5.
- ↔
RAG versus fine-tuning or a long context. RAG keeps knowledge fresh and citable and is cheap to update. Fine-tuning (training an existing model a bit more on your own examples, which changes its weights; Topic 16.13) changes behaviour, not facts reliably. Very long contexts can work for small corpora but cost more per question.
Done when you can
Done when you can draw the RAG pipeline and name each step.
Done when you can chunk text with overlap and explain the effect.
Done when you can build a prompt with numbered sources and verify citations.
Done when you can say what happens when nothing relevant is found.
Done when you know a full Hectal course on RAG and agents is planned.