A spot check passes while the golden set finds a regression
A fake model replies from a script that depends on the system prompt. The golden set and fake are pasted at the top; later examples reuse them.
LLM features answer differently on each run and change behaviour when a prompt or model changes, so a few manual tries prove almost nothing. An evaluation (an eval) is a fixed set of cases scored the same way every time, and it is the test suite of an LLM app.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write diff_runs(old, new) for two lists of booleans (one per case) that returns three lists of indexes: fixed (fail then pass), broken (pass then fail) and still failing.
Write pass_rate(results) that returns the fraction of True values as a number with 3 decimals, and all_three_pass(rate) that returns the chance that three independent tries all pass at that rate.
Explain it without notes
Why is manually trying a few inputs not enough to ship an LLM feature?
What is a silent regression and what causes one in an LLM app?
How is an eval different from a unit test?
Why can a prompt that raises the average score still be worse?
Expected output
spot check of prompt v2 (3 tickets):
t1: looks fine
t2: looks fine
t6: looks fine
golden set, prompt v1: 5 of 8 pass
golden set, prompt v2: 6 of 8 pass
fixed by v2: t5, t8
broken by v2: t3