Score retrieval and answers separately
The runs are logged data, not live calls, so the three languages print the same lines. The faithfulness check here is deliberately crude: Topic 9.5 improves it.
A RAG product is a search system feeding a writing system, and each can fail alone. Reading a few answers cannot tell which one broke, so this phase measures retrieval and answers separately, with numbers.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write classify(retrieval_ok, faithful) that returns one of four sentences: working, generator drifted, retrieval missed, fix search first. Print the result for all four input combinations.
Given the list of runs from the example, write count_outcomes(runs) that returns how many fall into each of the four outcomes, using retrieval_ok and faithful. Print the counts.
Explain it without notes
Why is a RAG system harder to evaluate than a plain model call?
Give an example of a retrieval failure and a generation failure, and say how each shows up in the logged record.
Why is a faithful answer not necessarily a correct one?
Why check retrieval before answer quality?
Expected output
What does the monthly pass cost?
retrieval: found D8 | answer: from the sources
Can I get my money back if I cancel?
retrieval: found D1 | answer: NOT in the sources
I forgot my login, how do I get back in?
retrieval: MISSED D3 | answer: from the sources
My tyre is flat, what now?
retrieval: found D10 | answer: from the sources
retrieval found the passage: 3/4
answer matches its sources: 3/4
both right: 2/4