Topic 16.11
Testing and Evaluating AI Code
In one line
Model output is not deterministic, but the code around it must be testable. Use fakes and recorded replies for unit tests, an eval set with scoring functions to measure quality, regression checks to catch drops, and logs of prompts and replies to debug what happened.
Think of it like this
A driving school. The instructor does not test a student by asking for an essay on driving. There is a fixed test route with fixed things to check: stops at the line, mirror before turning. A student can have a good day or a bad day, so the school counts across many routes. For AI code you need both: tests that check your own driving rules, and an eval that scores the model across many fixed cases.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Unit test
- A small automatic check that one piece of your code works.
- Fake
- A stand-in object with scripted behaviour, used instead of the real thing in a test.
- Recorded response
- A saved reply that a test replays instead of calling the real model.
- Eval (evaluation)
- Measuring how good the model's answers are on a fixed set of cases.
- Scoring function
- A function that decides how well an actual answer matches what was expected.
- Baseline
- The previous good score that new results are compared with.
- Regression
- A change that makes results worse than before.
Step by step
01Test your code with a recorded model
The first example replays three saved replies, including one off-script answer ("I think it is neutral."). The test checks that classify normalises case and whitespace and returns "unknown" for anything outside the allowed labels. The model never changes between runs, so the test never flakes.
02Build an eval set and scorers
The second example has four cases and two scorers. run_eval returns a pass rate and the failing prompts, so you know *what* failed, not only how many. The models here are plain dictionaries of canned answers, so every number is the same each time you run it. A scorer is just a function, so you can write exact, contains-all, JSON-valid or any custom rule.
03Compare against a baseline
"prompt v1" passes 3 of 4 and meets the baseline. "prompt v2" passes 2 of 4 and is flagged REGRESSION. In CI (continuous integration: a server that runs your tests on every change) this would fail the build and stop a worse prompt from shipping. The printed failures show which cases to read.
04Log what was sent and received
The wrapper class records one JSON line per call with the prompt version and length. Real logs also carry model name, token counts and latency, and are written with the logging module to a file or log service. Keys and personal data must be removed first.
05Do not assert on free text
The break-it example asserts that two phrasings of the same fact are equal. It fails even though both are right. Instead of asserting the exact text, assert a property: contains the key fact, is valid JSON, cites a real source, stays under a length.
06Testing injection defences
Treat Topic 16.7's defences like any other feature: add inputs that contain harmless injection-style phrases, and assert the behaviour you want (flagged in the log, no tool call made, answer unchanged). That turns a worry into a regression test.
Try it yourself
- 1
Fix prompt v2
In the eval example change the two wrong answers for prompt v2 so it passes all four. What pass rate prints, and does the status change?
- 2
Add a case
Add a fifth case where both prompts are wrong. How do the pass rates and the baseline comparison change?
Code & diagrams
Expected output
3 deterministic tests passed; model calls: 3The "models" are dictionaries of canned answers. Edit an answer and watch the pass rate change.
Expected output
prompt v1: pass rate 0.75 failures ['opposite of hot'] -> OK
prompt v2: pass rate 0.50 failures ['2+2', '3*3'] -> REGRESSION
True FalseExpected output
{"n": 1, "prompt_chars": 5, "prompt_version": "summary-v3", "reply": "HELLO"}
{"n": 2, "prompt_chars": 11, "prompt_version": "summary-v3", "reply": "SECOND CALL"}Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Assert the exact wording of a model reply
Test a model that is allowed to phrase the same answer differently.
replies = iter(["The answer is 4.", "It's 4!", "Four"])
fake_but_unstable = lambda prompt: next(replies)
assert fake_but_unstable("2+2") == "The answer is 4."
assert fake_but_unstable("2+2") == "4", "same prompt, different wording"Myth vs fact
Myth
AI features cannot be tested because the output varies.
Fact
Your code around the model is fully testable with fakes. The model's quality is measured with evals on a fixed set.
Myth
One good demo means it works.
Fact
A demo shows a few lucky cases. An eval set shows the pass rate across many, including the awkward ones.
Myth
Using a model as judge removes the need for human review.
Fact
Judges are useful for scale, but they can be wrong or biased. Sample their verdicts and check them by hand.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Separate retrieval metrics from answer metrics in RAG. "Was the right chunk in the top k?" is deterministic and cheap to compute, and tells you whether to fix chunking or prompts.
- ▸
Keep eval data versioned like code, and include the real failures from production (with personal data removed). A growing set of failures is the most valuable test asset a team has.
- ▸
Cost control for evals: cache model replies keyed by (model, prompt version, input), run a small subset on every commit and the full set nightly, and track the cost per eval run next to the pass rate.
- ▸
Temperature 0 reduces randomness but does not guarantee identical output on every call (batching, floating-point order and provider changes can still shift results), and some models do not accept a temperature setting at all. Design evals to tolerate noise: run each case several times for important checks, report the pass rate with its spread, and compare changes with enough cases that a one-case swing is not mistaken for progress. With only 20 cases, one case is 5 percentage points.
- ▸
A model used as a judge has known biases: it can prefer longer answers, prefer the answer shown first, and favour text written by itself. Reduce them with a clear rubric, scoring each answer separately where you can, randomising order for pairwise comparisons, and above all measure the judge: label a sample by hand and check how often the judge agrees with the human labels before you trust it on the rest. Keep a held-out set you do not tune the prompt against, or the score only measures how well the prompt was fitted to those cases.
Remember this
- 1
Two different questions. (1) Does *my code* work correctly, whatever the model says? That is a normal unit test (Topic 11.1). (2) Is the *model's answer good enough*? That is an evaluation (eval). Mixing them makes tests flaky and slow. Keep them apart.
- 2
Unit tests use fakes. The model is passed in as a parameter (Topic 16.3), so a test gives it a scripted function or a recorded model that replays saved replies. The test then checks your parsing, validation, retries, trimming and error handling deterministically, with no network, no cost and no randomness. A recorded model that raises on an unknown prompt tells you immediately when a prompt changed.
- 3
An eval set is a list of cases: an input and what a good answer looks like. A scoring function turns (expected, actual) into pass/fail or a number. Scorers can be exact match, "contains these facts", "is valid JSON with these keys", "the cited source number exists" (Topic 16.9), or, for open-ended text, another model acting as judge (used carefully: judges have their own biases and need spot checks by humans).
- 4
Regression checks. Keep the last good pass rate as a baseline. When you change a prompt, model or chunk size, run the eval and compare. A drop beyond a tolerance fails the check. Because model output can vary, run important cases several times or use temperature 0 where available (temperature is a setting for how random the model's word choices are, and 0 makes them as predictable as possible), and set the tolerance for noise rather than expecting identical numbers.
- 5
Log prompts and replies with the prompt template name and version, the model name, token counts and a request ID, and remove secrets and personal data. When a user reports a bad answer, the logged messages let you replay and turn the case into a new eval item. The wrapper in the third example does this in a few lines.
- 6
Evals cost money and time. Every case is a model call. Keep a small fast set for every change, a larger set before releases, and cache replies when only your scoring code changed. Also check the set itself: if it is small, or all cases are easy, a high score tells you little. Add every real failure you meet.
Explain it without notes
What is the difference between a unit test and an eval for AI code?
How do you test code that calls a non-deterministic model?
How does a regression check work for a prompt change?
You use another LLM to grade answers. How do you decide whether you can trust those grades, and how do you set up an eval pipeline for a RAG feature in CI?
Practice
Write contains_all(reply, words), a scorer that ignores case.
Write run_eval(model, cases, scorer) that returns the share of passing cases, and 0.0 for an empty list.
Write regression(baseline, current, tolerance) that says whether quality dropped by more than the tolerance.
Trade-offs
- ↔
Small fast eval versus large thorough eval. A small set runs on every change and catches big breaks. A large set finds subtle drops but costs time and money. Use both.
- ↔
Exact scorers versus model judges. Exact scorers are cheap, repeatable and narrow. Judges handle open-ended text and add cost and noise.
Done when you can
Done when you can explain the difference between a test and an eval.
Done when you can write a recorded or scripted fake model and test code that uses it.
Done when you can write an eval runner with swappable scoring functions.
Done when you can compare a pass rate to a baseline with a tolerance.
Done when you know a full Hectal course on LLM engineering is planned to cover evals in depth.