A rubric judge and a strict verdict parser
The judge is a fake with scripted replies, including three broken ones. The parser rejects them with a reason.
When no rule can decide whether an answer is good, you can ask another model to grade it against a written rubric. It scales, but it has biases (position, length, self-preference) that you must measure and calibrate against human grades.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write consistent_winner(first_run, second_run) where each argument is the winner ("A" or "B") from one order, with the second run already mapped back to the original labels. Return the winner when they agree and "tie" otherwise. Print the result for ("A", "A"), ("A", "B") and ("B", "B").
Write flip_rate(pairs) where pairs is a list of (first_run, second_run) winners. Return the fraction of pairs that flipped, rounded to 3 decimals. Print it for [("A", "A"), ("A", "B"), ("B", "A"), ("B", "B")].
Explain it without notes
When should you use an LLM judge instead of a rule-based scorer?
What is position bias and how do you measure and mitigate it?
Name two other judge biases and a defence for each.
How do you know your judge can be trusted?
Expected output
answer 1: score 5 (States the refund and the 5 day limit.)
answer 2: score 2 (No refund and no timing.)
answer 3: rejected (not JSON)
answer 4: rejected (score must be a whole number from 1 to 5)
answer 5: rejected (missing reason)
accepted 2 of 5, mean score 3.500
first prompt sent to the judge:
Score the answer from 1 (bad) to 5 (excellent). A good answer states the refund and the 5 business day limit. Reply with JSON that has the keys score and reason.
Question: How long do refunds take?
Answer: Refunds are issued within 5 business days of approval.