An eval set, scorers and a regression check
The "models" are dictionaries of canned answers. Edit an answer and watch the pass rate change.
Model output is not deterministic, but the code around it must be testable. Use fakes and recorded replies for unit tests, an eval set with scoring functions to measure quality, regression checks to catch drops, and logs of prompts and replies to debug what happened.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write contains_all(reply, words), a scorer that ignores case.
Write run_eval(model, cases, scorer) that returns the share of passing cases, and 0.0 for an empty list.
Write regression(baseline, current, tolerance) that says whether quality dropped by more than the tolerance.
Explain it without notes
What is the difference between a unit test and an eval for AI code?
How do you test code that calls a non-deterministic model?
How does a regression check work for a prompt change?
You use another LLM to grade answers. How do you decide whether you can trust those grades, and how do you set up an eval pipeline for a RAG feature in CI?
Expected output
prompt v1: pass rate 0.75 failures ['opposite of hot'] -> OK
prompt v2: pass rate 0.50 failures ['2+2', '3*3'] -> REGRESSION
True False