Topic 9.8
Playbook: An Eval Harness
In one line
The recipes of this phase in one place: a golden dataset, a runner, scorers, a report table, a diff between prompt versions and a CI gate with a threshold, a baseline and critical cases, all in Python, Java and JavaScript.
Think of it like this
A kitchen checklist taped to the wall. Before each service the cook checks the fridge temperature, the knives, the stock and the labels, in the same order every day, and the restaurant opens only when every line is ticked. The recipes below are the checklist for shipping a change to an LLM feature, and the harness is the clipboard.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Harness
- The code that runs your cases through the feature and scores them, the same way every time.
- Row
- One case's facts: id, wanted, got, and each check's result.
- Report
- The readable table and summary the harness prints or saves.
- Diff
- A comparison of two runs that lists fixed, broken and still-failing cases.
- Gate
- A check that decides whether a change may go ahead.
- Adapter
- A small class that gives a real client the same interface as the fakes.
Step by step
01Step 1: the dataset and the fake model
Both examples start with the same eight tickets and the same scripted model as the whole phase. In your project the model line is the only thing that changes between a unit test (a fake), a CI run (a recorder) and a nightly run (the real client).
02Step 2: one function returns rows
evaluate parses each reply, compares the fields and returns rows with id, want, got, valid, cat, pri and ok. Rows are plain data, so the report, the gate and the diff all read the same thing.
rows.append({"id": c["id"], "want": c["category"] + "/" + c["priority"], "got": shown,
"valid": valid, "cat": cat_ok, "pri": pri_ok, "ok": cat_ok and pri_ok})03Step 3: print the report
The table has one line per case, so a failing case is visible at once: t3 and t4 fail for prompt v2. The summary below it shows pass rate 0.750, category 0.875, priority 0.875 and valid JSON 1.000.
04Step 4: gate the change
The gate collects problems: a pass rate below 0.700 and any failed critical case (t1 and t2). With no problems the status is 0. Replace CANDIDATE with SYSTEM_V3 and t1 fails, so the gate blocks even though the average is the same.
05Step 5: diff against the baseline
The second example compares v1 with v2 and with v3. v2 fixes t5 and t8, breaks t3, and is allowed with a warning. v3 fixes the same two cases but breaks the critical case t1 and is blocked. This report, attached to the pull request, is what a reviewer actually reads.
06Step 6: plug in the real client
The last block shows an adapter class that calls a real model over plain HTTP, with the same complete(system, user, temperature) method as the fakes. Nothing else in the harness changes. Run it nightly with repeated runs, and keep the replay version for pull requests.
Try it yourself
- 1
Gate a different prompt
In the first example set
CANDIDATE = SYSTEM_V3. Predict the table and the gate line, then run. Which rule blocks it? - 2
Add a critical case
Add
"t3"toCRITICALand run the first example withSYSTEM_V2. What does the gate say now, and is that the right call for a customer asking how to change their email address? - 3
Diff v2 and v3
In the second example add
review("v2", SYSTEM_V2, "v3", SYSTEM_V3). Which case is fixed, which is broken? Is this change allowed, and does that match your own judgment? - 4
Write down the pipeline
List what runs on every pull request (replay), what runs nightly (live, repeated, with a judge for free text) and where the report is saved. Compare your list with the workflow in Topic 9.5.
Code & diagrams
Dataset, runner, scorer, report table, per-field rates and a CI gate in one program. The exit status is printed instead of exiting.
Expected output
id want got result
t1 billing/high billing/high PASS
t2 bug/high bug/high PASS
t3 account/low other/low FAIL
t4 billing/low billing/high FAIL
t5 bug/high bug/high PASS
t6 other/low other/low PASS
t7 account/low account/low PASS
t8 bug/low bug/low PASS
pass rate 0.750 | category 0.875 | priority 0.875 | valid JSON 1.000
gate: passed
exit status: 0Lists fixed, broken and still-failing cases. A broken critical case blocks the change even when the average does not drop.
Expected output
v1 -> v2
fixed: t5, t8
broken: t3
still failing: t4
pass rate 0.625 -> 0.750 (+0.125)
allowed (warning: 1 case got worse)
v1 -> v3
fixed: t5, t8
broken: t1
still failing: t4
pass rate 0.625 -> 0.750 (+0.125)
BLOCKED: critical case broken: t1Not run here: it needs your own GROQ_API_KEY and a network. Plain HTTP against an OpenAI-compatible endpoint. Java uses Jackson for JSON, which real projects use instead of the course helper.
# pip install httpx
import os
import httpx
class GroqModel:
"""The same complete(system, user, temperature) interface as the fakes, now calling a real model."""
def complete(self, system, user, temperature=0):
response = httpx.post(
"https://api.groq.com/openai/v1/chat/completions",
headers={"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]},
json={
"model": "openai/gpt-oss-20b",
"temperature": temperature,
"max_completion_tokens": 400, # a reasoning model spends hidden tokens first
"messages": [{"role": "system", "content": system}, {"role": "user", "content": user}],
},
timeout=30,
)
response.raise_for_status()
return response.json()["choices"][0]["message"]["content"]
# Pass GroqModel() where the examples passed a fake, or wrap it in the Recorder from Topic 9.6.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Leave the unreadable replies out of the denominator
In the first Python example set CANDIDATE = SYSTEM_V1, and change rate = sum(r["ok"] for r in rows) / n to divide by the number of rows whose JSON is valid: rate = sum(r["ok"] for r in rows) / sum(r["valid"] for r in rows).
Myth vs fact
Myth
An eval harness needs a framework.
Fact
The harness here is a short program in each language. Frameworks add dashboards and integrations, which are useful later, but the dataset, the scorers and the gate are the parts that matter.
Myth
Once the gate is green, the feature is good.
Fact
The gate says it has not got worse on the cases you thought of. Real traffic (Topic 9.7) and human review keep finding new cases.
Myth
I can copy someone else's eval set and skip building mine.
Fact
A public benchmark measures the model in general. Your feature's failures live in your own inputs and labels.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Version everything the score depends on: dataset commit, prompt text, model name, sampling settings, judge prompt. Store them with each result so that two numbers are only compared when the rest was the same.
- ▸
Keep the report diff-friendly (stable order, fixed decimals, no timestamps in the body). Then two reports can be compared with an ordinary text diff in a pull request.
- ▸
Run the gate in two stages: a cheap replay on every pull request and a repeated live run nightly. Page someone only for the nightly run, and open an issue automatically with the report attached.
- ▸
Review the harness itself now and then: sample ten passing and ten failing rows and check by hand that the scorer agrees with a person.
Remember this
- 1
Recipe: golden dataset in code. When to use it: always, before the first prompt tweak. Keep cases as a typed list in the repo with an id, the input, the labels, and a
typicaloredgetag. It guards against editing prompts blind. Adapt it by validating the file on every run (unique ids, labels from a fixed list) and by adding a case for every production failure (Topics 9.2 and 9.7). - 2
Recipe: runner with the model as a parameter. When to use it: for every feature.
evaluate(model_or_system)loops over the cases, calls the model through a small interface, and returns one row of facts per case, never a single number. It guards against hard-wired clients that cannot be faked or recorded. Adapt it by passing a fake, a recorder or the real client (Topic 9.6). - 3
Recipe: strict parse, then score fields separately. When to use it: whenever the reply is structured. Parse the whole reply as JSON (a reply in chatter is a failure), then compare each field on its own. It guards against hiding a format problem inside an average and tells you which field is weak (Topic 9.3). Adapt it by adding precision, recall and F1 per class when the classes are uneven, and a judge for free-text fields (Topic 9.4).
- 4
Recipe: a report you can read. When to use it: every run. Print one line per case (
id, wanted, got,PASSorFAIL), then the pass rate, the per-field rates and the valid-JSON rate. It guards against a bare score that nobody can act on. Adapt it by saving the same rows as a file that CI uploads as an artifact. - 5
Recipe: diff two prompt versions. When to use it: on every prompt or model change. Run the baseline and the candidate and list the cases fixed, broken and still failing, with the change in pass rate. It guards against silent regressions that an average hides (Topic 9.1). Adapt it by turning a broken critical case into a hard failure.
- 6
Recipe: the CI gate. When to use it: when the eval runs in a pipeline. The gate returns problems, not just a number: pass rate below the threshold, a drop below the baseline larger than the margin, a failed critical case. It ends with an exit status that CI understands (printed here instead of exiting). It guards against merging a worse prompt. Adapt it by repeating cases and taking a majority when scores are flaky, and by running a replay in pull requests and a live run nightly (Topic 9.5).
- 7
Recipe: record, replay and key by a hash. When to use it: to run the harness in CI for free and in milliseconds. Save each reply under a hash of everything that influences the answer and replay it later. A changed prompt changes the key and fails loudly. It guards against slow, costly and flaky CI. Adapt it by adding the model name and tool definitions to the key (Topic 9.6).
- 8
Recipe: judge with swapped orders. When to use it: for answers no rule can score. Ask a judge with a rubric, parse its verdict strictly, judge every pair in both orders and accept only agreement (Topic 9.4). It guards against position bias. Adapt it by calibrating against a human-graded sample.
Explain it without notes
Walk through the parts of an LLM eval harness and what each one is responsible for.
Why does the harness return rows rather than a single score?
What do you need to run the same harness in a unit test, in pull-request CI and nightly?
How would you decide that a prompt change is safe to merge?
Practice
Write summarise(rows) where each row is a dict with a boolean ok. Return a string such as 3 of 4 pass (0.750). Print it for rows with ok values True, True, False, True.
Write broken_cases(old_rows, new_rows) where each is a dict from case id to boolean. Return the sorted ids that passed before and fail now. Print it for old = {"a": True, "b": True, "c": False} and new = {"a": False, "b": True, "c": True}.
Trade-offs
- ↔
A small hand-built harness is easy to understand and change; a framework adds dashboards and integrations at the price of learning it.
- ↔
Running everything on every pull request is safest but slow and costly; split into replay on pull requests and live nightly.
- ↔
Many metrics describe quality better, but one headline number plus critical cases makes a release decision easier.
Done when you can
Before you ship: the golden set is validated, tagged typical or edge, and includes this month's production failures.
Before you ship: the eval runs on every prompt, model and parsing change, with a replay in pull requests and a live run nightly.
Before you ship: the gate has a threshold, a baseline margin and a list of critical cases, and ends with an exit status.
Before you ship: scores are repeated or have a known noise level, so a red build means something.
Before you ship: judges are calibrated against human grades and orders are swapped.
Before you ship: the report shows fixed, broken and still-failing cases, and every broken case has been looked at.
Before you ship: production feedback and a reviewed sample flow back into the golden set.