Diff two prompt versions and gate on the regression
Lists fixed, broken and still-failing cases. A broken critical case blocks the change even when the average does not drop.
The recipes of this phase in one place: a golden dataset, a runner, scorers, a report table, a diff between prompt versions and a CI gate with a threshold, a baseline and critical cases, all in Python, Java and JavaScript.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write summarise(rows) where each row is a dict with a boolean ok. Return a string such as 3 of 4 pass (0.750). Print it for rows with ok values True, True, False, True.
Write broken_cases(old_rows, new_rows) where each is a dict from case id to boolean. Return the sorted ids that passed before and fail now. Print it for old = {"a": True, "b": True, "c": False} and new = {"a": False, "b": True, "c": True}.
Explain it without notes
Walk through the parts of an LLM eval harness and what each one is responsible for.
Why does the harness return rows rather than a single score?
What do you need to run the same harness in a unit test, in pull-request CI and nightly?
How would you decide that a prompt change is safe to merge?
Expected output
v1 -> v2
fixed: t5, t8
broken: t3
still failing: t4
pass rate 0.625 -> 0.750 (+0.125)
allowed (warning: 1 case got worse)
v1 -> v3
fixed: t5, t8
broken: t1
still failing: t4
pass rate 0.625 -> 0.750 (+0.125)
BLOCKED: critical case broken: t1