Four rule checks on five answers
The answers are fixed data standing in for a model's output, so the three languages print the same lines. Real projects usually add rules for formatting and banned phrases too.
Once retrieval is measured, the answer needs checking too: is it faithful to the sources, does it address the question, and did the product refuse when it should? Cheap rule checks come first, a judge model second.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write numbers_ok(answer, source) that returns True when every number in the answer (after removing [D1] style markers) also appears in the source. Print it for two cases.
Write parse_judge(reply) that returns (faithful, relevant) as ints or None when the reply is not JSON with both keys. Print the result for a good reply and for the text fine.
Explain it without notes
What are the three properties of a good RAG answer, and which need extra information to check?
Name four rule checks and one thing rule checks cannot see.
What is a rubric and why does the judge reply as JSON?
How would you check that a judge model is trustworthy?
Expected output
case 1: cites ok numbers ok refusal ok length ok | What does the monthly pass cost?
case 2: cites ok numbers FAIL refusal ok length ok | Can I get my money back if I cancel?
case 3: cites FAIL numbers ok refusal ok length ok | Is a helmet included?
case 4: cites FAIL numbers FAIL refusal FAIL length ok | Do you rent scooters?
case 5: cites ok numbers ok refusal ok length ok | Can I bring my dog on the bike?
passed: cites 3/5 numbers 3/5 refusal 4/5 length 5/5