Topic 9.4
LLM-as-Judge
In one line
When no rule can decide whether an answer is good, you can ask another model to grade it against a written rubric. It scales, but it has biases (position, length, self-preference) that you must measure and calibrate against human grades.
Think of it like this
A talent show with one judge who has favourites. She likes the act that performs first, gives extra marks to long acts, and loves anyone who sings like her. You can still use her scores, but only after you notice the habits, shuffle the order of acts, and check her against a second judge. An LLM judge is that judge.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- LLM-as-judge
- Using a language model to grade another model's answer against written instructions.
- Rubric
- Written grading rules that say what each score means.
- Pairwise comparison
- Showing a judge two answers and asking which is better.
- Position bias
- A judge's habit of preferring the answer shown in a certain position.
- Length bias
- A judge's habit of preferring longer answers.
- Self-preference bias
- A model's habit of rating its own (or its family's) output higher.
- Calibration
- Checking a judge against human grades and adjusting until they agree enough.
- Agreement
- The share of cases where two graders give the same verdict.
Step by step
01Step 1: write the rubric
Decide what a good answer to the question "How long do refunds take?" contains: the refund and the 5 business day limit. Write that in the rubric, with what each score means, and ask for JSON with score and reason.
Score the answer from 1 (bad) to 5 (excellent).
A good answer states the refund and the 5 business day limit.
Reply with JSON that has the keys score and reason.02Step 2: build the prompt and call the judge
The prompt is the rubric, then the question, then the answer. The first example uses a FakeJudge with scripted verdicts, so you can see how the code handles good, bad and broken replies. It also records every prompt it was sent, so a test can check the prompt.
03Step 3: never trust the verdict format
The judge is a model: it sometimes replies Score: 4 out of 5, returns a score of 9, or leaves the reason empty. parse_verdict rejects all three with a reason, and the harness counts rejected verdicts (here 3 of 5) instead of averaging garbage. Retry a rejected verdict once, then mark the case as ungraded (Topic 6.8).
04Step 4: compare two answers
For "which prompt is better", show the judge answer A and answer B for the same input. Pair every case twice: A first, then B first. The second example does this for six pairs against a judge that favours the first position by 15 points.
05Step 5: read the flips
Pairs p1, p2, p3 and p6 flipped: their true qualities differ by less than 15, so the bias decides. p4 and p5 (gaps of 35 and 35) are stable. A judge that flips on 4 of 6 pairs is telling you almost nothing about close calls, and close calls are exactly what you compare when you tweak a prompt.
06Step 6: calibrate and keep it honest
Grade a sample by hand, compare, and keep the human-graded sample as a permanent check on the judge. Use the judge for trends and for ranking, and keep a human in the loop for decisions that matter. The labs below let you write a judge prompt for a real model and then test it for position bias.
Try it yourself
- 1
Predict the flips
In the second example change
BIASto 5. Before running, decide which pairs will still flip (the ones with a quality gap under 5). Then run and check. - 2
Remove the bias
Set
BIASto 0. How many pairs flip now? What does that tell you about a fair judge on close pairs (look at thefaircolumn)? - 3
Break the parser
Add a sixth reply to the first example:
{"score": 4.5, "reason": "Close."}. Predict whether it is accepted, run all three languages, and decide what the rule should be for a half point. - 4
Use the labs
In the first lab, make the answer worse (remove the word
quick) and the rubric stricter. In the second, swap the answers and send three times. Write down how often the letter stayed the same.
Code & diagrams
The judge is a fake with scripted replies, including three broken ones. The parser rejects them with a reason.
Expected output
answer 1: score 5 (States the refund and the 5 day limit.)
answer 2: score 2 (No refund and no timing.)
answer 3: rejected (not JSON)
answer 4: rejected (score must be a whole number from 1 to 5)
answer 5: rejected (missing reason)
accepted 2 of 5, mean score 3.500
first prompt sent to the judge:
Score the answer from 1 (bad) to 5 (excellent). A good answer states the refund and the 5 business day limit. Reply with JSON that has the keys score and reason.
Question: How long do refunds take?
Answer: Refunds are issued within 5 business days of approval.The fake judge adds a fixed bonus to whichever answer comes first. Judging both orders and accepting only agreement removes the bias.
Expected output
p1 A first: A B first: B flipped fair: tie better: A
p2 A first: A B first: B flipped fair: tie better: B
p3 A first: A B first: B flipped fair: tie better: A
p4 A first: A B first: A stable fair: A better: A
p5 A first: B B first: B stable fair: B better: B
p6 A first: A B first: B flipped fair: tie better: A
verdict flipped when the order was swapped: 4 of 6 pairsTry: Edit the prompt so a real model grades the answer against your rubric. Make it return JSON with the keys score (1 to 5) and reason. Try changing the rubric wording, then send again and see how the score changes. Does the model obey the format every time?
Goals
- The verdict has the keys score and reason
- The score is a whole number from 1 to 5
Sign in to send this request to a real model (free).
Try: Send this once and note the winner. Then swap the two answers (put the text of A under B and the other way round), send again, and compare. If the winner letter stays the same while the text behind it changed, the judge followed the position, not the answer. Try it a few times with temperature 0.
Goals
- The verdict has the keys winner and reason
- The winner is A or B
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Trust the first order only
In the second Python example, change the line fair = w1 if w1 == w2 else "tie" to fair = w1, so the harness uses only the verdict from the A-first run.
Myth vs fact
Myth
A judge model is objective.
Fact
It is a model with habits: it prefers the first answer, the long answer and its own style. Measure those habits before relying on it.
Myth
If a strong model judges, no human grading is needed.
Fact
A judge needs a human-graded sample to calibrate against, and a fresh check whenever the judge model or the rubric changes.
Myth
A bigger judge model fixes bias.
Fact
Bigger judges are usually better at following a rubric, but position and length bias have been reported for many judge models. Swap orders and test your own judge rather than assume.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Cost: a judge call is another model call per case, often with a long prompt (the question, the answer, the rubric, sometimes reference documents). Run the judge only on cases rules cannot score, and use a smaller judge for broad runs and a stronger one for final decisions.
- ▸
For faithfulness checks (does the answer only use the documents it was given, Topic 8.6), give the judge the sources and ask for the unsupported claims, not a single number. Verdicts that point at text can be checked by a person quickly.
- ▸
Reference-guided judging is more reliable than open judging: give the judge a known-good answer and ask whether the candidate says the same thing. When you have a golden answer, prefer this.
- ▸
Version the judge like code: the model name, the prompt and the temperature. A silent change in the judge looks exactly like a change in your feature's quality.
Remember this
- 1
Why this exists. Scorers from Topic 9.3 work when a rule can decide. But "is this explanation clear?", "is this summary faithful to the article?" and "which of these two replies is more helpful?" need judgment. Humans give the best judgment but are slow and expensive. An LLM-as-judge sends the question, the answer and a rubric (written grading instructions) to a model and asks for a verdict. It is fast, cheap and tireless, and it is wrong in predictable ways.
- 2
Write a rubric, not a vibe. "Rate the answer from 1 to 5" gives noise. "5 = states the refund and the 5 business day limit; 3 = states one of them; 1 = states neither" gives repeatable grades. Ask for a short reason next to the score, and a strict output format (JSON with
scoreandreason) so code can read it. Use temperature 0 and always validate the verdict, as you validated structured output in Topic 4.5 (the first example rejects a score of 9, a missing reason and a reply that is not JSON). - 3
Pairwise comparison. Instead of an absolute score, show two answers (A and B) and ask which is better. Comparing two answers is usually easier, and more consistent, than rating one on a scale, and the result tells you directly whether the new prompt beats the old one. The cost is that a pairwise judge has position bias: it tends to prefer the answer shown first (or, for some models, second), even when the answers are equal.
- 4
Measure position bias by swapping. Show A then B, then B then A, and count how often the verdict follows the position rather than the answer. In the second example a judge adds 15 points to the answer it reads first. When two answers are within 15 points of each other, the verdict flips when the order is swapped: 4 of 6 pairs. The fix is to judge both orders and count a result only when both agree, otherwise call it a tie.
- 5
Other biases. Length bias: judges often prefer the longer answer, even when it just pads. Self-preference: a model tends to rate text written by itself or by its own family higher. Style bias: confident, well-formatted answers beat correct but plain ones. Defences: judge with a different model family than the one that wrote the answers, control for length in the rubric ("do not reward length"), and test the judge on pairs where you know which answer is better.
- 6
Calibrate against humans. Before you trust a judge, have people grade 30 to 50 answers, run the judge on the same answers, and measure agreement (the share of answers with the same verdict, or the difference in score). If agreement is low, fix the rubric and try again. Re-check after any change to the judge model or prompt. A judge is a measuring instrument: it needs calibrating and it drifts.
Explain it without notes
When should you use an LLM judge instead of a rule-based scorer?
What is position bias and how do you measure and mitigate it?
Name two other judge biases and a defence for each.
How do you know your judge can be trusted?
Practice
Write consistent_winner(first_run, second_run) where each argument is the winner ("A" or "B") from one order, with the second run already mapped back to the original labels. Return the winner when they agree and "tie" otherwise. Print the result for ("A", "A"), ("A", "B") and ("B", "B").
Write flip_rate(pairs) where pairs is a list of (first_run, second_run) winners. Return the fraction of pairs that flipped, rounded to 3 decimals. Print it for [("A", "A"), ("A", "B"), ("B", "A"), ("B", "B")].
Trade-offs
- ↔
Judges scale to open-ended answers but cost tokens and carry their own errors; rules are free and exact but narrow.
- ↔
Judging both orders doubles the cost and removes position bias; judging one random order halves the cost and needs more pairs.
- ↔
A strong judge agrees better with humans but costs more and is slower; use it for decisions and a small one for daily runs.
Done when you can
Done when you can write a rubric whose scores have concrete meanings.
Done when you can parse and validate a judge's JSON verdict and count rejected ones.
Done when you can measure position bias by swapping A and B and counting flips.
Done when you can name length and self-preference bias and a defence for each.
Done when you can describe how to calibrate a judge against human grades.