Topic 8.5
A Working RAG Pipeline
In one line
A RAG pipeline is four small functions in a row: ingest the documents once, then for each question retrieve the best chunks, build a prompt with numbered sources, and let the model answer. Here the whole thing runs by hand in three languages.
Think of it like this
A research assistant in a library. Before the semester, they read every book once and file index cards (ingest). When you ask a question, they fetch the two or three best cards (retrieve), paper-clip them to your question (augment), and hand it to the professor, who writes the answer (generate). The professor never walks the shelves.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Pipeline
- A fixed series of steps where each step's output is the next step's input.
- Ingest
- Load, chunk, embed and store documents ahead of time.
- Retrieve
- Find the chunks most relevant to a question.
- Augment
- Add the retrieved chunks to the prompt.
- Generate
- Let the model write the answer from the prompt.
- Minimum score
- A cut-off below which a retrieved chunk is dropped as not relevant enough.
- Prompt template
- The fixed text with blanks that your code fills in for every question.
Step by step
01Step 1: ingest once
Turn each document into a chunk with a number, a source name, its text and its vector. This is the only step that touches all the documents, and it runs when they change.
chunks = [
{"number": n, "source": name, "text": text, "vec": embed(text)}
for n, (name, text) in enumerate(DOCS, start=1)
]02Step 2: retrieve for each question
Embed the question, score every chunk, sort by score with the document number as tie-breaker, keep the top 2 and drop scores under 0.3. The program prints what it retrieved before it prints anything else, so you can see what the model will see.
03Step 3: build the prompt
Instructions, numbered sources, question. The first question prints the whole prompt with a | in front of each line, so you can read exactly what a real model would be sent.
04Step 4: generate, and the empty case
The fake model copies source [1] with its number. For the question with no retrieved sources, the prompt has an empty source list and the fake model answers "I do not know." A real model should do the same because the template tells it to, but you will check that in Topic 8.6 rather than hope.
05Step 5: debug in order
When an answer is wrong, look at the stages in order: were the right chunks retrieved? Was the prompt built correctly? Did the model use the sources? Fix the first stage that is wrong. Starting with the model is the most common mistake.
Try it yourself
- 1
Change k
Set
kto 3 for the first question and predict what the third retrieved chunk is (look at the payment document's cost hit and its score of 0.1715 before the minimum score is applied). Does it appear? Why not? - 2
Change the minimum
Raise the minimum score to 0.6. Which sources survive for "how much is express delivery"? Lower it to 0.1. What shows up now?
- 3
Reorder the sources
Swap the order in which
build_promptlists sources (put the weakest first). The fake model copies [1]. What does that tell you about why real models also care about source order?
Code & diagrams
The fake model copies the first numbered source. Real models write their own answer, but the plumbing around them is the same.
Expected output
question: how much is express delivery
retrieved: [1] shipping 1.0000, [2] helmets 0.5000
| Answer using only the sources below. Cite them like [1]. If they do not contain the answer, say you do not know.
|
| Sources:
| [1] Standard shipping takes 3 to 5 days. Express shipping takes 1 to 2 days and costs 12 dollars.
| [2] Every bike comes with a free helmet. Extra helmets cost 25 dollars.
|
| Question: how much is express delivery
answer: Standard shipping takes 3 to 5 days. Express shipping takes 1 to 2 days and costs 12 dollars. [1]
question: get my money back
retrieved: [1] returns 1.0000
answer: Unused bikes can be returned within 30 days with the receipt for a full refund. [1]
question: do you sell locks
retrieved: nothing
answer: I do not know.Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
No minimum score
In the example, change the default min_score=0.3 in retrieve to min_score=0.0.
Myth vs fact
Myth
RAG is a product you install.
Fact
It is a pattern: retrieve, augment, generate. Frameworks wrap these functions, but the same four steps are underneath, and you can write and test them yourself.
Myth
If the answer is wrong, the model is the problem.
Fact
Usually retrieval is. Print the retrieved chunks first; if the right chunk is missing no model can fix it.
Myth
More retrieved chunks always give better answers.
Fact
Extra chunks add cost and distraction, and can contradict each other. Tune k with test questions.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log every stage: question, retrieved chunk ids and scores, the final prompt and the model's reply. Without this you cannot debug a bad answer after the fact (Phase 11).
- ▸
Ingest and query paths must share the exact same embedder and tokenisation. A common bug is re-embedding documents with a new model while queries still use the old one.
- ▸
Latency adds up: embedding the question, searching, then the model call. The model call dominates for hosted models; retrieval from memory takes milliseconds, from a remote database tens of milliseconds.
- ▸
Retrieved documents are untrusted input. Keep your instructions outside them, and never give the model tools just because a document asked for them (Topic 10.2).
Remember this
- 1
Why this exists. Topics 8.2 to 8.4 gave you the parts. A feature needs them wired together in a known order, with clear places to log, test and replace each step. Building the pipeline by hand first shows that RAG is not magic or a framework: it is four ordinary functions, and every bug lives in exactly one of them.
- 2
Step 1, ingest (once, ahead of time). Load documents, cut them into chunks, embed each chunk and store text, vector and metadata. Here each of the eight documents is one chunk. In production this runs when documents change, not on every question, and it records the embedding model used.
- 3
Step 2, retrieve (every question). Embed the question with the same embedder, score every chunk by cosine similarity, sort best first with a tie-breaker, keep the top k, and drop anything below a minimum score. In the example k = 2 and the minimum is 0.3. The minimum matters: for "do you sell locks" the question contains no concept word, every score is 0, so nothing is retrieved.
- 4
Step 3, augment (build the prompt). Instructions first (answer only from the sources, cite like [1], say so if the answer is missing), then the sources numbered from 1, then the question. Numbering is not decoration: it gives the model a short handle to cite and gives your code something to check (Topic 8.6). Always keep the instructions in your own template, never inside the retrieved text.
- 5
Step 4, generate. Send the prompt to the model. The fake model here is a few lines: it finds source [1] and copies it with its number. That is on purpose, so you can see the plumbing and test everything else with no network. A real model reads all sources, picks the relevant facts and writes its own sentence. Everything before this step is plain code you can unit-test.
- 6
Where it breaks. Wrong retrieval gives a wrong or irrelevant answer in a confident voice: for "how much is express delivery" the second source is the helmets document (score 0.5000), which is noise; a real model usually ignores it, but it still costs tokens. No retrieval means the model gets an empty source list and (if it follows instructions) says it does not know, which is why the instructions line matters. When an answer is wrong, print the retrieved chunks first: most of the time the fault is there, not in the model.
Explain it without notes
Name the four steps of a RAG pipeline and say which ones run once and which run per question.
Why should you inspect the retrieved chunks before blaming the model for a wrong answer?
What is the purpose of a minimum score in retrieval?
Why number the sources in the prompt?
Practice
Write build_prompt(question, chunks) where chunks is a list of strings. It returns the lines Sources:, then [1] ..., [2] ..., then a blank line and Question: ..., joined with newlines. Print it for two sources.
Write retrieve(scores, k, min_score) where scores is a list of (name, score) pairs. Return the names of the best k with score at least min_score; equal scores are ordered by name.
Trade-offs
- ↔
A higher k gives more evidence and more noise and cost; a lower k is cheaper and may miss the supporting chunk.
- ↔
A strict minimum score avoids irrelevant context but raises the number of questions answered with "not in the documents".
- ↔
Embedding at ingest makes queries fast but needs a re-ingest when documents or the embedding model change.
Done when you can
Done when you can write ingest, retrieve, build_prompt and answer as four separate functions.
Done when you can print and read the exact prompt the model receives.
Done when you can explain what happens when retrieval returns nothing.
Done when you can debug a wrong answer by inspecting the stages in order.