A judge with a rubric (scripted model)
fake_judge replies from a script, so this runs offline. In Java the course's Json helper parses the reply; real Java projects use Jackson (or Gson), and this course's Json keeps examples runnable in the browser.
Once retrieval is measured, the answer needs checking too: is it faithful to the sources, does it address the question, and did the product refuse when it should? Cheap rule checks come first, a judge model second.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write numbers_ok(answer, source) that returns True when every number in the answer (after removing [D1] style markers) also appears in the source. Print it for two cases.
Write parse_judge(reply) that returns (faithful, relevant) as ints or None when the reply is not JSON with both keys. Print the result for a good reply and for the text fine.
Explain it without notes
What are the three properties of a good RAG answer, and which need extra information to check?
Name four rule checks and one thing rule checks cannot see.
What is a rubric and why does the judge reply as JSON?
How would you check that a judge model is trustworthy?
Expected output
Score 1 to 5. faithful: every claim is supported by the sources. relevant: the answer addresses the question. Reply with JSON only: {"faithful": n, "relevant": n, "reason": "..."}
Question: What does the monthly pass cost?
Sources: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.
Answer: It costs 25 dollars and includes unlimited rides of up to 45 minutes.
case 1: faithful 5 relevant 5 PASS | Both facts match the source.
case 2: faithful 1 relevant 5 FAIL | The source says 2 hours, not 4.
case 3: judge error (reply was not the JSON we asked for)
passed 1/3, judge errors counted as not passed