Command Palette

Search for a command to run...

Hectal
Phase 9Advanced10 of 15 in LLM Engineering

Evaluation and Testing

How to know your LLM feature works, and keep knowing it: golden sets, scorers, judges, regression gates in CI, fakes, recordings and production feedback.

A normal program either passes its tests or it does not. An LLM feature answers differently from run to run, changes behaviour when you edit one sentence of the prompt, and changes again when the provider updates the model. "I tried it three times and it looked fine" is how quality quietly falls apart. This phase teaches the discipline that replaces it: evaluation (measuring quality on a fixed set of cases) and testing (checking the code around the model).

You build one eval harness by hand across the phase, around one small feature: sorting support tickets into a category and a priority. First a golden set of labelled cases, then scorers (exact match, JSON validity, per-field accuracy, precision, recall and F1), then a runner that compares prompt versions case by case. A scripted "prompt v2" visibly fixes two cases and silently breaks a third. That regression is what the phase is about.

Then the harness goes to work: an LLM judge for answers no rule can score (and its biases), a CI gate with a threshold and an exit status, fakes and recorded responses so most tests are free and repeatable, and online evaluation from real traffic. Examples run in Python, Java and JavaScript against fake models. Two real-model labs let you write a judge prompt and see position bias for yourself.

0/8 · 0%
8 topics ~3 h 21 code blocks & diagrams
Start with the first topic
1
9.1

Why "It Looked Fine" Fails

LLM features answer differently on each run and change behaviour when a prompt or model changes, so a few manual tries prove almost nothing. An evaluation (an eval) is a fixed set of cases scored the same way every time, and it is the test suite of an LLM app.

26 min 2 code practice

2
9.2

Golden Datasets

A golden dataset is the answer key for your LLM feature: real inputs with labels a human agreed on. How you collect it, which edge cases you add, how big it is and how you keep it fresh decides whether your eval tells the truth.

24 min 2 code practice

3
9.3

Scoring Functions

A scorer turns one model reply into a pass or a number. Exact match, contains, regex, JSON validity, field accuracy, precision, recall, F1 and numeric tolerance cover most checks, and each is a few lines of code you can write yourself.

25 min 2 code practice

4
9.4

LLM-as-Judge

When no rule can decide whether an answer is good, you can ask another model to grade it against a written rubric. It scales, but it has biases (position, length, self-preference) that you must measure and calibrate against human grades.

26 min 4 code practice

5
9.5

Regression Tests in CI

A regression test for an LLM feature runs the eval on every prompt, model or code change and fails the build when quality drops below a threshold or a critical case breaks. Repeating cases and judging by majority keeps random noise from blocking good changes.

26 min 4 code practice

6
9.6

Fakes and Recorded Responses

Most tests of an LLM feature should not call a real model. A fake model with scripted replies tests your code (parsing, retries, fallbacks, prompt assembly), and recorded real responses replayed by a request hash test it against realistic data, for free and in milliseconds.

23 min 2 code practice

7
9.7

Evaluating in Production

A golden set only covers the cases you thought of. Online evaluation learns from real traffic: user feedback, a stable sample of conversations for human review, and A/B tests that compare two prompts or models on live users while respecting statistical noise.

22 min 2 code practice

8
9.8

Playbook: An Eval Harness

The recipes of this phase in one place: a golden dataset, a runner, scorers, a report table, a diff between prompt versions and a CI gate with a threshold, a baseline and critical cases, all in Python, Java and JavaScript.

28 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Write two scorers

Write exact_match(want, got): True when the two strings are equal after removing spaces at both ends and ignoring upper and lower case, otherwise False.

Intermediate
2

Precision, recall and F1 for one class

Write precision_recall_f1(gold, pred, label). gold and pred are lists of labels of the same length. Count, for label: true positives (gold and pred both equal label), false positives (pred equals label, gold does not) and false negatives (gold equals label, pred does not).

Intermediate
3

Diff two eval runs

Write compare(old, new). old and new are dicts from case id to True (passed) or False (failed), with the same ids.

Intermediate
4

Write the CI gate

Write gate(results, threshold, critical). results is a dict from case id to True or False, threshold is the minimum pass rate, and critical is a list of case ids that must pass.

Advanced
5

Record and replay model calls

Complete the Recorder class. complete(prompt) computes key = tiny_hash(prompt). In "record" mode it calls self.live.complete(prompt), stores the reply in self.tape[key] and returns it. In "replay" mode it returns self.tape[key] and never calls the live model; if the key is not in the tape it raises KeyError.

Advanced
6

A judge that survives swapping

Write fair_verdict(judge, answer_a, answer_b). judge(first, second) returns "first" or "second": the position it prefers.

Advanced