PHASE 9Advanced ~27 min· topic 5 of 10

Topic 9.5

Judging the Answer

In one line

Once retrieval is measured, the answer needs checking too: is it faithful to the sources, does it address the question, and did the product refuse when it should? Cheap rule checks come first, a judge model second.

Think of it like this

Marking an essay. First the quick checklist anyone can apply: name written? within the word limit? did it quote a book from the reading list? Then a teacher reads it for sense and fairness. The checklist is fast, free and never changes its mind, but it can't read. The teacher can read, but costs time and has moods. Good marking uses both, checklist first.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Faithful (grounded): Every claim in the answer is supported by the passages that were sent.
Relevant: The answer addresses the question that was asked.
Correct: The answer matches the truth. Needs a reference answer or a person to check.
Rule check: A check written as plain code, such as 'every number in the answer appears in the source'.
Judge model: A model that scores another model's output against a rubric.
Rubric: Written instructions saying exactly what each score means.
Judge error: A judge reply that cannot be used, such as text that is not the JSON you asked for.
Calibration: Comparing the judge's scores with human labels to learn how far to trust it.

Step by step

01Step 1: write the rule checks

Each rule is a small function from (question, sources, answer) to yes or no. The four here need no model. Keep each one separate so a failure names the rule.

rules.pywhole filepython

def check(answerable, sent, answer):
    is_refusal = answer == REFUSAL
    cited = re.findall(r"\[(D[0-9]+)\]", answer)
    ...
    return {
        "cites": is_refusal or (len(cited) > 0 and all(d in sent for d in cited)),
        "numbers": all(n in evidence_numbers for n in claimed),
        "refusal": is_refusal == (not answerable),
        "length": len(answer.split()) <= 25,
    }

02Step 2: score unanswerable questions differently

For an unanswerable question the right answer is the refusal sentence, so the refusal rule fails an invented answer even when its other properties look fine. Case 4 shows it: it cites D8, a passage that was never sent.

03Step 3: write the judge prompt

The rubric names the two scores, the scale and the reply format. The user part holds the question, the sources and the answer. Keep the sources exactly as the generator saw them; judging against the whole handbook would excuse answers that went beyond what was sent.

Step 3: write the judge promptdiagram

Rendering diagram…

04Step 4: parse defensively

A judge is a model, so its reply may not be JSON, may miss a key or may contain a score outside the scale. The example catches the parse failure, returns nothing, and counts the case as 'judge error'. Counting an unparseable reply as a pass would hide the worst cases.

05Step 5: read the scripted results

Case 1 scores 5 and 5: pass. Case 2 scores faithful 1 with the reason 'The source says 2 hours, not 4': fail, and the reason is exactly what a developer needs. Case 3's reply is 'Looks fine to me.', which is not JSON: judge error. The run passes 1 of 3.

06Step 6: try a real judge

The lab sends the wrong-number case to a real model with the rubric. The goals check that it returns the JSON keys and that it gives faithful a low score. Reasoning is cheap to ask for and often useful: read it, and decide whether you would have agreed.

Try it yourself

1
Fix case 2
Change the answer of case 2 to Cancel at least 2 hours before the start for a full refund. [D1] and run. Which rule passes now? Does anything else change?
2
Add a rule
Add a fifth rule no_hedging that fails if the answer contains probably or I think. Add a case that trips it and read the new totals.
3
Change the scripted judge
In the judge example, make the second case's reply {"faithful": 4, "relevant": 5, "reason": "close enough"}. Does the case pass? What does that tell you about the pass mark and about trusting one number?
4
Use the lab
Send the lab, then change the temperature to 1 and send three times. Does the faithful score stay the same? Why is 0 the right setting for a judge?

Code & diagrams

Four rule checks on five answers New tab

The answers are fixed data standing in for a model's output, so the three languages print the same lines. Real projects usually add rules for formatting and banned phrases too.

Expected output

case 1: cites ok  numbers ok  refusal ok  length ok  | What does the monthly pass cost?
case 2: cites ok  numbers FAIL  refusal ok  length ok  | Can I get my money back if I cancel?
case 3: cites FAIL  numbers ok  refusal ok  length ok  | Is a helmet included?
case 4: cites FAIL  numbers FAIL  refusal FAIL  length ok  | Do you rent scooters?
case 5: cites ok  numbers ok  refusal ok  length ok  | Can I bring my dog on the bike?
passed: cites 3/5  numbers 3/5  refusal 4/5  length 5/5

A judge with a rubric (scripted model) New tab

fake_judge replies from a script, so this runs offline. In Java the course's Json helper parses the reply; real Java projects use Jackson (or Gson), and this course's Json keeps examples runnable in the browser.

Expected output

Score 1 to 5. faithful: every claim is supported by the sources. relevant: the answer addresses the question. Reply with JSON only: {"faithful": n, "relevant": n, "reason": "..."}
Question: What does the monthly pass cost?
Sources: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.
Answer: It costs 25 dollars and includes unlimited rides of up to 45 minutes.
case 1: faithful 5  relevant 5  PASS  | Both facts match the source.
case 2: faithful 1  relevant 5  FAIL  | The source says 2 hours, not 4.
case 3: judge error (reply was not the JSON we asked for)
passed 1/3, judge errors counted as not passed

A real model as the judgereal model

Try: Send the request. The answer says 4 hours, the source says 2. Does the judge give faithful a low score and name the wrong number? Also read the relevant score: in one real run it was 1, though the answer does address the question, which is a reminder that judges need calibrating. Then change the answer to say 2 hours and send again: do the scores change? Try the answer 'Helmets are good for your health.' for the helmet question and see what the judge does with an answer that is true-sounding but off topic.

System

user

Modelmid-size, also reads imagesTemperature · 0.0Max tokens (1–1024) Stream (SSE) JSON mode

Goals

The reply is JSON with faithful, relevant and reason
faithful is 1 or 2
The reason mentions the real number (2)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Count a judge error as a pass

In the Python tab of the judge example, add passed += 1 just before the continue in the judge-error branch, so an unusable reply counts as a pass. Run.

terminal

$ Run

── what you'll see ──

Score 1 to 5. faithful: every claim is supported by the sources. relevant: the answer addresses the question. Reply with JSON only: {"faithful": n, "relevant": n, "reason": "..."}

Question: What does the monthly pass cost?

Sources: The monthly pass costs 25 dollars and includes unlimited rides of up to 45 minutes.

Answer: It costs 25 dollars and includes unlimited rides of up to 45 minutes.

case 1: faithful 5 relevant 5 PASS | Both facts match the source.

case 2: faithful 1 relevant 5 FAIL | The source says 2 hours, not 4.

case 3: judge error (reply was not the JSON we asked for)

passed 2/3, judge errors counted as not passed

Myth vs fact

Myth

A judge model's score is the truth.

Fact

It is another model's opinion. Compare it with human labels on a sample before you trust it, and re-check when the judge model changes.

Myth

If an answer cites a source it is faithful.

Fact

The citation can be real and the claim still wrong. Check that the cited text supports the sentence, not just that the id exists.

Myth

Rule checks are too simple to be useful.

Fact

They catch the most common clear failures for free, deterministically, on every commit. Use them first and add a judge for what they cannot see.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Judge each property separately with a small scale (for example 1 to 5 or pass and fail). One vague 'quality' score is hard to act on and hard to calibrate.
▸
Use the judge's reason as data: group failing reasons ('wrong number', 'adds a claim', 'off topic') to find the most common cause, then fix it once for many questions (Topic 9.6).
▸
Pairwise judging ('is answer A better than B?') is often more reliable than absolute scores when comparing two versions, but it needs both orders to cancel position bias.
▸
Keep a few unanswerable rows in every judged run. A judge that never sees a refusal can't tell you whether refusals are correct.

Remember this

1
Why answers need their own checks. Retrieval metrics (9.3 and 9.4) say whether the right passage arrived. They say nothing about what the model did with it. Topic 9.1 showed an answer that changed a number and one that was faithful to the wrong passage. People can't read hundreds of answers after every change, so the checks have to run by themselves.
2
Three properties. Faithful: every claim in the answer is supported by the passages that were sent. Relevant: the answer addresses the question that was asked. Correct: the answer matches the truth, which needs a reference answer or a person. Faithfulness and relevance can be checked from the question, the sources and the answer alone. Correctness needs more, so it is checked on the smaller set of rows that carry a reference answer.
3
Rule checks first. Code can check a lot without a model. Does the answer cite a source that was actually sent (not an invented id)? Does every number in the answer appear in the cited sources? Did an unanswerable question get the fixed refusal sentence, and did an answerable question avoid it? Is the answer short enough? The first example runs these four on five answers: citing passes 3 of 5, numbers 3 of 5, refusal 4 of 5, length 5 of 5. Case 2 said 4 hours where the source says 2; case 3 had no citation; case 4 invented a scooter price for a question the handbook can't answer and failed three rules at once.
4
What rules miss. A rule can check that '25' appears in the source, not that the sentence uses it correctly. It sees no meaning: 'Helmets are good for your health' passes the numbers rule and does not answer the question. Rules are narrow, precise and cheap, so they catch the common, clear failures, and the rest goes to a judge.
5
A judge model. A second model reads the question, the sources and the answer and scores them against a rubric: written instructions that say exactly what each score means. Ask for the scores as JSON (faithful, relevant, a one-line reason) so your code can read them. The second example shows the prompt and parses three scripted replies: a clean pass, a fail where the reason names the wrong number, and a reply that is not JSON at all. That last one is a judge error: count it separately and never as a pass.
6
Judges have biases. They tend to prefer longer answers, answers that appear first when two are compared, and answers written by themselves or models like them (LLM Engineering Topic 9.4 covers this). Reduce the risk: use a specific rubric with a small scale, set the temperature to 0, judge each answer alone against the sources, and where you can use a different model from the one being judged. Calibrate: have a person label 30 answers, run the judge on the same ones, and measure how often they agree. A judge nobody has compared with a human is a guess with a confident voice.
7
Cost and where it fits. A judge is one extra model call for every answer scored, so run it on the whole golden set only when something important changes, and on a sample otherwise. Rule checks run on every commit. Topic 8.7 checks groundedness at answer time for the user; this topic is the offline measurement of the same property over a fixed set, so you can compare versions.

Explain it without notes

What are the three properties of a good RAG answer, and which need extra information to check?

Name four rule checks and one thing rule checks cannot see.

What is a rubric and why does the judge reply as JSON?

How would you check that a judge model is trustworthy?

Practice

Write numbers_ok(answer, source) that returns True when every number in the answer (after removing [D1] style markers) also appears in the source. Print it for two cases.

Write parse_judge(reply) that returns (faithful, relevant) as ints or None when the reply is not JSON with both keys. Print the result for a good reply and for the text fine.

Trade-offs

↔
Rule checks: fast, free and exact, but narrow. Judge model: broad and flexible, but costly, variable and biased.
↔
Judging every answer: complete numbers, high cost. Judging a sample: cheaper, noisier.
↔
A strict pass mark (all scores 4 or more): few bad answers pass, more good ones fail on a judge's mood. A loose one: the opposite.

Done when you can

Done when you can define faithful, relevant and correct and say which needs a reference.
Done when you can write four rule checks and say what they miss.
Done when you can write a judge prompt with a rubric and parse its reply safely.
Done when you can describe how to calibrate a judge against humans.

Command Palette

Words you'll meet

Step by step

01Step 1: write the rule checks

02Step 2: score unanswerable questions differently

03Step 3: write the judge prompt

04Step 4: parse defensively

05Step 5: read the scripted results

06Step 6: try a real judge

Try it yourself

Fix case 2

Add a rule

Change the scripted judge

Use the lab

Code & diagrams

Break it on purpose

Count a judge error as a pass

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can