Position bias: swap A and B and count the flips
The fake judge adds a fixed bonus to whichever answer comes first. Judging both orders and accepting only agreement removes the bias.
When no rule can decide whether an answer is good, you can ask another model to grade it against a written rubric. It scales, but it has biases (position, length, self-preference) that you must measure and calibrate against human grades.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write consistent_winner(first_run, second_run) where each argument is the winner ("A" or "B") from one order, with the second run already mapped back to the original labels. Return the winner when they agree and "tie" otherwise. Print the result for ("A", "A"), ("A", "B") and ("B", "B").
Write flip_rate(pairs) where pairs is a list of (first_run, second_run) winners. Return the fraction of pairs that flipped, rounded to 3 decimals. Print it for [("A", "A"), ("A", "B"), ("B", "A"), ("B", "B")].
Explain it without notes
When should you use an LLM judge instead of a rule-based scorer?
What is position bias and how do you measure and mitigate it?
Name two other judge biases and a defence for each.
How do you know your judge can be trusted?
Expected output
p1 A first: A B first: B flipped fair: tie better: A
p2 A first: A B first: B flipped fair: tie better: B
p3 A first: A B first: B flipped fair: tie better: A
p4 A first: A B first: A stable fair: A better: A
p5 A first: B B first: B stable fair: B better: B
p6 A first: A B first: B flipped fair: tie better: A
verdict flipped when the order was swapped: 4 of 6 pairs