Topic 9.1
Why "It Looked Fine" Fails
In one line
LLM features answer differently on each run and change behaviour when a prompt or model changes, so a few manual tries prove almost nothing. An evaluation (an eval) is a fixed set of cases scored the same way every time, and it is the test suite of an LLM app.
Think of it like this
A teacher who marks an exam by reading the first three answers on the pile. Those three look good, so she gives the whole class a pass. The mistakes are in answers 4 to 30. A proper marking scheme reads every paper with the same checklist, and also lets her compare this year's class with last year's. An eval is that marking scheme for your model feature.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Eval (evaluation)
- Running a fixed set of cases through your feature and scoring the replies the same way every time.
- Non-determinism
- The same input can give different outputs on different runs.
- Temperature
- A setting that controls how random the model's word choices are. 0 is the least random.
- Golden set
- A list of inputs paired with the correct (or acceptable) outputs, used as the answer key.
- Regression
- Something that used to work and no longer does, after a change.
- Silent regression
- A regression that raises no error, so only a comparison with known-good results shows it.
- Scorer
- A function that turns one model reply into a score or a pass/fail.
Step by step
01Step 1: the feature and the change
The feature reads a support ticket and answers with JSON: a category (billing, bug, account or other) and a priority (high or low). The team rewrites the system prompt from v1 to v2 to make the model answer with JSON only. The author tries three tickets by hand, and all three look right.
02Step 2: write the answer key
Eight tickets, each with the category and priority a human agreed on. This is the golden set (Topic 9.2). It lives in code, next to the feature, in version control.
GOLDEN = [
{"id": "t1", "ticket": "I was charged twice for my March invoice", "category": "billing", "priority": "high"},
{"id": "t3", "ticket": "How do I change my email address?", "category": "account", "priority": "low"},
# ... six more
]03Step 3: score every case
A case passes when the reply is valid JSON and both fields match the key. The runner sends each ticket through the fake model with prompt v1, then with prompt v2, and records pass or fail per case. A reply like Sure! Here is the JSON: {...} fails because it is not valid JSON, even though the content is right (Topic 4.2).
04Step 4: compare the versions
Overall, v2 scores higher: 6 of 8 against 5 of 8. It fixed two cases (t5 and t8). But case t3 passed with v1 and fails with v2. The total went up while the product got worse for everyone who asks how to change their email address. Only the per-case comparison shows this.
05Step 5: one case is a sample of one
The second example makes the model noisy: it answers wrongly one time in four, decided by a small seeded random generator so the output is repeatable. Five developers who try the ticket once each can see a failure or not. Twenty runs reveal the real pass rate. Quality has to be measured as a rate over many runs and many cases, not read from one reply.
06Step 6: the habit
Before any prompt, model or code change ships: run the eval, look at fixed and broken cases, and decide. The rest of this phase automates that habit (Topic 9.5) and extends it to answers a rule cannot score (Topic 9.4) and to live traffic (Topic 9.7).
Try it yourself
- 1
Predict, then run
Before running the first example, add
t3toSPOT_CHECK. Predict the first block of output, then run it. Which ticket would have warned the author? - 2
Add a case that breaks v1
Add a ninth ticket to
GOLDENand a ninth reply to each list in the model part. Make v1 answer it wrongly and v2 correctly. The fixed list should now name three cases. - 3
Change the noise
In the second example, change
0.75to0.9innoisy_modelonly. Does the pass rate over 20 runs match 0.9? Why not exactly? Then change the startingstateto 7 and compare.
Code & diagrams
A fake model replies from a script that depends on the system prompt. The golden set and fake are pasted at the top; later examples reuse them.
Expected output
spot check of prompt v2 (3 tickets):
t1: looks fine
t2: looks fine
t6: looks fine
golden set, prompt v1: 5 of 8 pass
golden set, prompt v2: 6 of 8 pass
fixed by v2: t5, t8
broken by v2: t3The seeded generator (state * 48271 % 2147483647) makes the noise repeatable and identical in all three languages.
Expected output
five developers try the ticket once each: pass, pass, pass, pass, pass
the same ticket 20 times: 15 pass, 5 fail
pass rate: 0.750
chance that 3 manual tries all pass: 0.422Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Judge only by the total
At the end of the first Python example, delete the two lines that print fixed by v2 and broken by v2, so the program prints only the two totals.
Myth vs fact
Myth
If I test it ten times and it works, it works.
Fact
Ten tries on inputs you chose, with a prompt you tuned for them, measure how well you tuned. A fixed set that includes cases you did not tune on, scored by code, is the minimum.
Myth
Temperature 0 makes the output deterministic, so one run is enough.
Fact
Hosted models can still vary slightly between runs, and a prompt edit or model update changes the answer for every temperature. Evals compare across versions, not just across runs.
Myth
A higher average score means the new prompt is better.
Fact
The average hides regressions. A prompt that fixes two rare cases and breaks a common one raises the average and lowers the quality that users feel. Look at which cases changed, and weight them by how often real users send them.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Treat the eval as a product artifact. It lives in the repo, runs in CI, has an owner, and every production bug becomes a new case before the fix is merged (the same habit as a bug-reproducing unit test).
- ▸
Pin and record the model version in every eval result. "Quality dropped" means two different things when the prompt changed and when the provider's
latestalias moved. Compare like with like. - ▸
Decide per case how much it matters. A critical case (the refund request that must not be misrouted) is a hard gate; a nice-to-have case is part of an average. Topic 9.5 builds both.
- ▸
An eval is itself code that can be wrong: a bad answer key or a loose scorer gives false confidence. Review a sample of passing cases by eye now and then.
Remember this
- 1
Why this exists. In ordinary code, the same input gives the same output, so a unit test that passes today passes tomorrow. A model call does not work that way. The answer can change from run to run, a one-line prompt edit can fix one case and break another, and the provider can update the model under you. Without a fixed set of cases, scored the same way each time, you cannot tell whether a change made the feature better, worse, or just different.
- 2
Non-determinism means the same input can give different outputs. With temperature above 0 (the setting that controls how adventurous the model's word choices are, Topic 1.6) the model picks among likely next words at random. Even at temperature 0 a hosted model can differ slightly between runs. So one try is a sample of size one. If a case passes 75% of the time, three manual tries all pass about 42% of the time, and you would ship a feature that fails every fourth user.
- 3
A silent regression is a quality drop that raises no error. The call succeeds, the JSON parses, nothing crashes, but some inputs that used to work are now wrong. Prompt edits cause them (Topic 3.8), and so do model upgrades and retired model versions. Because nothing crashes, only a test that compares today's answers with known-good answers can catch it.
- 4
The golden set is a list of inputs paired with the answers you want (Topic 9.2). A scorer turns one model reply into a number or pass/fail (Topic 9.3). An eval run sends every case through the feature and scores each reply, giving per-case results you can compare between two versions of your prompt: which cases it fixed, which it broke, and which still fail. The example below does exactly that for ticket triage.
- 5
Evals are not unit tests, and they do not replace them. Unit tests check your code with a fake model and must always pass (Topic 9.6). Evals check the model's behaviour on realistic input and give a score that moves up and down, so you set a threshold and watch trends (Topic 9.5). You need both, plus a look at real traffic (Topic 9.7).
- 6
Start small. Eight labelled cases and a 20-line runner already beat intuition. Teams that wait for the perfect dataset never start. Add a case every time a real user finds a failure, and the set becomes the memory of everything that ever went wrong.
Explain it without notes
Why is manually trying a few inputs not enough to ship an LLM feature?
What is a silent regression and what causes one in an LLM app?
How is an eval different from a unit test?
Why can a prompt that raises the average score still be worse?
Practice
Write diff_runs(old, new) for two lists of booleans (one per case) that returns three lists of indexes: fixed (fail then pass), broken (pass then fail) and still failing.
Write pass_rate(results) that returns the fraction of True values as a number with 3 decimals, and all_three_pass(rate) that returns the chance that three independent tries all pass at that rate.
Trade-offs
- ↔
Bigger golden sets give steadier numbers but cost time to label and money to run; start with a few dozen cases.
- ↔
Running the eval on every change catches regressions early but costs tokens; use a small fast set on every commit and the large set nightly (Topic 9.5).
- ↔
Strict scorers are repeatable but reject good answers worded differently; loose scorers accept bad answers. Topic 9.3 covers choosing.
Done when you can
Done when you can explain why one manual try proves little for a non-deterministic feature.
Done when you can define a silent regression and name two causes.
Done when you can run two prompt versions on a golden set and list fixed, broken and still-failing cases.
Done when you can say why an average score alone is not enough to approve a change.