Topic 0.1
What RAG Is
In one line
Retrieval-augmented generation (RAG) means your program first finds the passages that answer a question and then asks a language model to answer using those passages. It turns a closed-book exam into an open-book one.
Think of it like this
Two students sit the same exam. The first must answer from memory. The second may bring the textbook, find the right page and then write the answer in their own words. The second student makes fewer mistakes, can answer about chapters added last week, and can say which page the answer came from. RAG gives a language model the textbook.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- RAG (retrieval-augmented generation)
- Finding relevant text first, adding it to the prompt, then letting the model write the answer from it.
- Retrieval
- Searching your own documents for the few pieces that matter for one question.
- Index
- Your documents, prepared in advance so that searching them is fast.
- Document
- One source of text: a page, a file, an article.
- Chunk (passage)
- A piece of a document, small enough to put in a prompt.
- Query
- What you search with. Usually the user's question, sometimes a rewritten version of it.
- Context
- The passages you place in the prompt for the model to read.
- Grounded
- An answer is grounded when every claim in it comes from the passages provided.
Step by step
01Step 1: ask a model about something it never saw
Lumen Bikes is a made-up bike-rental company. Its handbook is private, so no model was trained on it. Ask a model "When do the shops open?" and it has two choices: admit it doesn't know, or guess. Models are trained to be helpful, so they often guess, in a confident voice.
02Step 2: find the page first
Before calling the model, our program searches the handbook for the passage that best matches the question. The simplest possible search counts shared words: "shops" and "open" both appear in the opening-hours passage, so it wins.
def retrieve(question):
wanted = set(tokenize(question))
best, best_score = None, 0
for doc_id, text in HANDBOOK:
score = len(wanted & set(tokenize(text))) # words in both
if score > best_score:
best, best_score = (doc_id, text), score
return best03Step 3: put the page in the prompt
The passage is pasted into the prompt, above the question, with an instruction such as "answer using only the source". The model now reads the fact instead of recalling it. This is the *augmented* part.
Answer using only the source.
Source [D2]: Shops open at 7 am and close at 9 pm every day. On public holidays they close at 6 pm.
Question: When do the shops open?04Step 4: the model writes the answer and names the source
The model answers from the passage and adds the passage's id, [D2]. That id is a citation: the reader (or your code) can check the answer against the source. A closed-book answer can never offer that.
05Step 5: notice what did the work
The model did the easy part: rewording a sentence it was handed. The hard part was choosing the right passage. If retrieval had returned the helmet rules, the model would have answered about helmets, or made something up. The quality of a RAG system is mostly the quality of its retrieval. That is why this course spends its first seven phases there.
Try it yourself
- 1
Ask about helmets
Change the question to
"Is a helmet included?". Predict which passage is retrieved and what the open-book answer will be, then run. - 2
Add a passage
Add
("D9", "Items left on a bike are kept at the nearest shop for 14 days.")to the handbook and ask"How long are lost items kept?". No retraining, no new model: the answer changes because the index changed. - 3
Compare the languages
Switch between the Python, Java and JavaScript tabs. Find the line in each that counts the shared words. All three print the same output.
Code & diagrams
The fake model stands in for a real one: with no source it answers from a tiny memory (and guesses when the memory is empty); with a source it answers from the source and cites it.
Expected output
Question: When do the shops open?
Closed book: Probably 9 am, like most shops. (a guess: the handbook was never in my training text)
Retrieved: D2 (Shops open at 7 am and c...)
Open book: Shops open at 7 am and close at 9 pm every day. [D2]
General knowledge still works: Paris.Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Ask in words the handbook doesn't use
Change the question to the same question in German, "Wann machen die Laeden auf?", and run the Python tab.
Myth vs fact
Myth
RAG trains the model on your documents.
Fact
Nothing is trained. The passages are pasted into one prompt and forgotten after the call. Change a document and the very next answer changes.
Myth
RAG stops hallucination.
Fact
It reduces it by giving the model facts to read. A model can still misread a passage, mix two passages, or answer from memory when retrieval returns the wrong text.
Myth
RAG is an AI technique.
Fact
Two of its three parts are search and data preparation. The teams with the best RAG systems are usually the ones with the best search.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Think of RAG as two systems with separate failure modes: retrieval (did the right passage reach the prompt?) and generation (did the model use it faithfully?). Always debug in that order. Phase 9 measures each one separately.
- ▸
The retriever sets a ceiling. If the answer is in the top 5 passages for 70 percent of questions, no prompt or model upgrade gets you past 70 percent correct, grounded answers.
- ▸
RAG shifts cost from training to every single request: the retrieved passages are input tokens you pay for on each call (LLM Engineering Topic 6.1). Topic 0.3 does the sums.
Remember this
- 1
Why this exists. A large language model (LLM) answers from what it absorbed during training. It has never seen your company's handbook, your product's documentation or yesterday's price change, and it cannot look anything up. Ask it anyway and it produces likely-sounding text, which may be wrong. Engineers needed a way to give the model the right facts at the moment of the question, without retraining it. That way is retrieval.
- 2
The name, word by word. *Retrieval* means finding the few pieces of text that matter for this question, out of everything you have. *Augmented* means added to: those pieces are added to the prompt. *Generation* is the model writing the answer. So RAG is: find, add, write. The model itself is unchanged. All the work happens in ordinary code around it.
- 3
Three parts. Every RAG system, from the toy on this page to a product serving millions, has the same three parts. An index: your documents, prepared ahead of time so they can be searched quickly. A retriever: given a question, it returns the best few passages from the index. A generator: the model, given the question plus those passages, writes the answer. Two of the three parts are search, not AI. Most RAG bugs live there.
- 4
The words you will meet everywhere. A document is one source: a page, a PDF, an article. A chunk (or passage) is a piece of a document small enough to hand to the model. The query is what you search with, usually the user's question. The context is the set of passages you put in the prompt. An answer is grounded when everything in it comes from that context and not from the model's memory.
- 5
What RAG does not do. It does not make the model smarter, and it does not teach the model anything permanently: the passages are forgotten after the call. It cannot answer from a passage it failed to find. And it does not stop a model from ignoring or misreading a passage it was given. RAG moves the problem from *what the model knows* to *what your search finds*, which is a much better problem to have, because search can be measured and fixed (Phase 9).
- 6
Where this course goes. LLM Engineering Phase 8 (
/llm-engineering/phase/phase-8) builds a small RAG in one phase. This course spends thirteen more phases on the same pipeline, because each part hides real engineering: keyword search and search that understands meaning (Phases 1 to 3), preparing documents (Phase 4), storing vectors (Phase 5), better ranking and better questions (Phases 6 and 7), answers you can trust (Phase 8), measuring (Phase 9), security (Phase 10), running it for real (Phase 11), advanced patterns (Phase 12) and interviews with a final capstone (Phase 13). Topic 0.6 shows the whole map.
Explain it without notes
Explain RAG to someone who has never programmed, in two or three sentences.
Name the three parts of a RAG system and say what each one does.
Why can a RAG system answer a question about a document that was written this morning, when the model was trained a year ago?
"We added RAG, so the bot can't make things up any more." What is wrong with this claim?
Practice
Write retrieve(question, docs) that returns the id of the passage sharing the most different words with the question, or None when no passage shares any word. Test it on the three handbook passages with "Is a helmet included?" and "Do you sell gift cards?".
Write open_book_prompt(question, doc_id, text) that returns the three-line prompt from Step 3, and print it for the opening-hours passage.
Trade-offs
- ↔
Closed book: one call, no search to build or run, but the model knows nothing private or recent and gives no sources.
- ↔
Open book (RAG): answers from your current documents with sources, but you now own a search system, and every question costs extra input tokens.
- ↔
More passages in the prompt: a better chance that the answer is in there, but more cost, more latency and more text for the model to get lost in.
Done when you can
Done when you can explain RAG as "find, add, write" and as an open-book exam.
Done when you can name the index, the retriever and the generator and say which ones are search.
Done when you can define document, chunk, query, context and grounded.
Done when you can give two things RAG does not fix.