Topic 9.1
Two Systems, Two Ways to Fail
In one line
A RAG product is a search system feeding a writing system, and each can fail alone. Reading a few answers cannot tell which one broke, so this phase measures retrieval and answers separately, with numbers.
Think of it like this
A restaurant sends out the wrong dish. Either the waiter wrote the wrong order on the ticket, or the cook made the right order badly. If all you do is taste the dish, you cannot tell which one went wrong, and fixing the wrong person changes nothing. A RAG product has the same two people: the retriever (the waiter) and the generator (the cook).
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Retriever
- The part of a RAG system that finds passages for a question.
- Generator
- The model that writes the answer from the question and the retrieved passages.
- Faithful (grounded)
- An answer that says only what its sources say.
- Golden set
- Test questions with known right answers (here: the passages that answer them). Topic 9.2 builds one.
- Metric
- A number that scores one property of the system, such as how often the right passage is found.
- Silent failure
- A wrong result with no error and a normal-looking shape. Only measurement catches it.
- Stage
- One step of the pipeline: indexing, retrieval, prompt budget, generation.
Step by step
01Step 1: write down what a run produced
For each question keep four things: the question, the passage that should answer it, the ids the retriever returned and the answer the model wrote. This is the smallest record that lets you judge both stages. Phase 11 logs the same four things in production (Topic 11.6).
02Step 2: score retrieval from ids alone
Retrieval is right when the expected id is among the returned ids. No model is called. In the example, 3 of 4 runs pass, and the login question is the miss.
def retrieval_ok(expected, retrieved):
return expected in retrieved03Step 3: score faithfulness against the sources that were sent
The model only had the retrieved passages, so judge the answer against exactly those, not against the whole handbook. The example uses the crudest honest check: the answer's sentence must appear word for word in a retrieved passage. Topic 9.5 builds better rule checks and a judge model.
def faithful(answer, retrieved):
sentence = answer[: answer.rindex(" [")]
return any(sentence in HANDBOOK[doc_id] for doc_id in retrieved)04Step 4: read the four outcomes
Retrieval right and answer faithful: working. Retrieval right and answer unfaithful: the generator drifted, so look at the prompt and the model. Retrieval wrong and answer faithful: the generator did its job on bad input, so look at search. Both wrong: fix search first, because the answer score is meaningless until the right passage arrives.
05Step 5: notice what one number hides
Two of four runs are fully right, which sounds like 50 percent. But the other two have different causes and different fixes. A single end-to-end score tells you the product is bad; the two stage scores tell you where to work.
06Step 6: keep the record forever
Store runs like these in version control or a database. A bad answer reported by a user becomes a new row. The set grows exactly where the system is weak, which is the idea behind Topic 9.2.
Try it yourself
- 1
Predict the summary
Before running, read the four runs and write down how many pass retrieval, how many are faithful and how many are both. Then run and compare. Which question surprised you?
- 2
Fix one failure
Change the retrieved list of the login question to
["D3"]and the answer to the real password sentence plus[D3]. How do the three summary lines change? This is what a successful fix looks like in the numbers. - 3
Add a fifth run
Add a run whose retrieval misses and whose answer is the refusal sentence
"I can't find that in the handbook.". What does the faithful check say about it, and is that a fair verdict? (Topic 9.5 handles refusals properly.)
Code & diagrams
The runs are logged data, not live calls, so the three languages print the same lines. The faithfulness check here is deliberately crude: Topic 9.5 improves it.
Expected output
What does the monthly pass cost?
retrieval: found D8 | answer: from the sources
Can I get my money back if I cancel?
retrieval: found D1 | answer: NOT in the sources
I forgot my login, how do I get back in?
retrieval: MISSED D3 | answer: from the sources
My tyre is flat, what now?
retrieval: found D10 | answer: from the sources
retrieval found the passage: 3/4
answer matches its sources: 3/4
both right: 2/4Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Judge only that a citation exists
In the Python tab, replace the last line of faithful with return answer.endswith("]"), so it only checks that the answer ends with a citation. Run.
Myth vs fact
Myth
If the answers look right, the system is fine.
Fact
Fluent text looks right whether or not it is. Faithful-but-wrong answers and drifting answers both look fine on a quick read.
Myth
One overall accuracy score is enough.
Fact
A single score cannot tell you whether to fix search or the prompt. Score the stages separately.
Myth
A bad answer means the model needs replacing.
Fact
Often retrieval handed it the wrong passage. Check the ids first; it costs nothing.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Retrieval metrics are deterministic and cheap, so they can run on every commit. Answer metrics that call a model cost money and vary a little, so they usually run on a smaller schedule (Topic 9.7).
- ▸
A faithful answer is not a correct answer. It is correct only if the sources are correct and relevant. That is why you need both a retrieval check and an answer check, and why stale documents show up in neither (Topic 11.2).
- ▸
Measure the stages in order of cost: ids first, rule checks second, a judge model last. Most real failures are caught before the expensive step.
Remember this
- 1
Why evaluate at all. Phases 1 to 8 gave AskDocs many knobs: chunk size, the embedding model, the index, hybrid search, re-ranking, the prompt, the refusal bar. Every knob can help one question and hurt another. Without numbers, the only test is whoever tried the last demo, and that is how a change that sounds clever ships and quietly makes the product worse. Evaluation turns each change into a measured question: did it help, on which questions, and by how much?
- 2
Failure one: retrieval misses. The passage that answers the question never reaches the model, because it was not indexed, not found, ranked too low, or cut to fit the budget. The model still writes something, usually fluent, cited and wrong, using the passage it was given. Phase 0 showed this: the login question retrieved the refunds passage, and the answer was confident and false.
- 3
Failure two: generation drifts. The right passage is in the prompt, but the answer changes a number, adds a claim the passage never made, or ignores the passage and answers from the model's memory. Retrieval was perfect, the product is still wrong. A model that is faithful (also called grounded) says only what its sources say.
- 4
Why 'it looked right' fails. Reading an answer tells you whether it sounds right, not whether it is right, and fluent text sounds right even when it is not. You also rarely read the sources behind the answer. A person checking ten answers by eye will miss the ones that fail quietly, which are the ones that hurt users. The example below shows an answer that is faithful to the wrong passage: nothing in the text gives it away.
- 5
Measure the two stages separately. Retrieval needs no model at all: for each test question you know which passage should come back, so you compare ids. It runs in milliseconds, costs nothing and gives the same number every time. Answer quality needs more: rule checks first (is the citation real, do the numbers match the source), then a judge model for what rules cannot see. Always check retrieval first: if the right passage was never retrieved, no answer score can tell you why the answer is wrong.
- 6
What the example shows. Four logged runs. Retrieval found the right passage in 3 of 4. The answer matched its sources in 3 of 4. Both were right in only 2 of 4, and the failures are different questions: one answer drifted from a good source, another was faithful to a bad one. One number for the whole system would have hidden that, and the fix for each is in a different place.
- 7
The map of this phase. A golden set of questions with known answers (9.2). Retrieval metrics: recall, precision and hit rate (9.3), then rank-aware MRR and nDCG (9.4). Answer checks (9.5). A triage that names the broken stage (9.6). Regression tests so a bad change cannot merge (9.7). Signals from real users (9.8). Then the AskDocs harness (9.9) and the recipes (9.10). LLM Engineering Phase 9 teaches the general idea of evals for any model app; this phase goes deeper on what is specific to retrieval.
Explain it without notes
Why is a RAG system harder to evaluate than a plain model call?
Give an example of a retrieval failure and a generation failure, and say how each shows up in the logged record.
Why is a faithful answer not necessarily a correct one?
Why check retrieval before answer quality?
Practice
Write classify(retrieval_ok, faithful) that returns one of four sentences: working, generator drifted, retrieval missed, fix search first. Print the result for all four input combinations.
Given the list of runs from the example, write count_outcomes(runs) that returns how many fall into each of the four outcomes, using retrieval_ok and faithful. Print the counts.
Trade-offs
- ↔
Separate stage metrics: more numbers to read and keep, in return for knowing which part to fix.
- ↔
Cheap id-based checks first: fast and deterministic, blind to answer quality, so they never replace answer checks.
- ↔
Reading answers by hand: catches what no metric was written for, does not scale, and misses quiet failures. Use it to find new test cases, not as the test.
Done when you can
Done when you can name the two stages of a RAG system and one way each can fail.
Done when you can explain why a faithful answer can still be wrong.
Done when you can say what four things a run record must contain.
Done when you can explain why retrieval is checked before answers.