One try is a sample of size one
The seeded generator (state * 48271 % 2147483647) makes the noise repeatable and identical in all three languages.
LLM features answer differently on each run and change behaviour when a prompt or model changes, so a few manual tries prove almost nothing. An evaluation (an eval) is a fixed set of cases scored the same way every time, and it is the test suite of an LLM app.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write diff_runs(old, new) for two lists of booleans (one per case) that returns three lists of indexes: fixed (fail then pass), broken (pass then fail) and still failing.
Write pass_rate(results) that returns the fraction of True values as a number with 3 decimals, and all_three_pass(rate) that returns the chance that three independent tries all pass at that rate.
Explain it without notes
Why is manually trying a few inputs not enough to ship an LLM feature?
What is a silent regression and what causes one in an LLM app?
How is an eval different from a unit test?
Why can a prompt that raises the average score still be worse?
Expected output
five developers try the ticket once each: pass, pass, pass, pass, pass
the same ticket 20 times: 15 pass, 5 fail
pass rate: 0.750
chance that 3 manual tries all pass: 0.422