The harness: evaluate, report, gate, status
Dataset, runner, scorer, report table, per-field rates and a CI gate in one program. The exit status is printed instead of exiting.
The recipes of this phase in one place: a golden dataset, a runner, scorers, a report table, a diff between prompt versions and a CI gate with a threshold, a baseline and critical cases, all in Python, Java and JavaScript.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write summarise(rows) where each row is a dict with a boolean ok. Return a string such as 3 of 4 pass (0.750). Print it for rows with ok values True, True, False, True.
Write broken_cases(old_rows, new_rows) where each is a dict from case id to boolean. Return the sorted ids that passed before and fail now. Print it for old = {"a": True, "b": True, "c": False} and new = {"a": False, "b": True, "c": True}.
Explain it without notes
Walk through the parts of an LLM eval harness and what each one is responsible for.
Why does the harness return rows rather than a single score?
What do you need to run the same harness in a unit test, in pull-request CI and nightly?
How would you decide that a prompt change is safe to merge?
Expected output
id want got result
t1 billing/high billing/high PASS
t2 bug/high bug/high PASS
t3 account/low other/low FAIL
t4 billing/low billing/high FAIL
t5 bug/high bug/high PASS
t6 other/low other/low PASS
t7 account/low account/low PASS
t8 bug/low bug/low PASS
pass rate 0.750 | category 0.875 | priority 0.875 | valid JSON 1.000
gate: passed
exit status: 0