Per-field accuracy, precision, recall and F1 by hand
Scores prompt v2's replies on the golden set. Every number can be checked with a pencil: tp, fp and fn are counted per class.
A scorer turns one model reply into a pass or a number. Exact match, contains, regex, JSON validity, field accuracy, precision, recall, F1 and numeric tolerance cover most checks, and each is a few lines of code you can write yourself.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write prf(tp, fp, fn) returning precision, recall and F1, each rounded to 3 decimals, with 0.0 when a denominator is 0. Print the result for tp=3, fp=1, fn=2.
Write rel_close(want, got, tolerance) that returns True when got is a number within tolerance (as a fraction of want) of want, and False for anything that is not a number. Print the result for (100, "104", 0.05), (100, "106", 0.05) and (100, "abc", 0.05).
Explain it without notes
Define precision, recall and F1 for one class and say what each is good at revealing.
Why can accuracy be misleading, and what do you report instead?
When would you use exact match, contains, regex and JSON validity?
Why does the scorer itself need tests?
Expected output
category accuracy: 7/8 = 0.875
priority accuracy: 7/8 = 0.875
whole object right: 6/8 = 0.750
billing tp=2 fp=0 fn=0 precision=1.000 recall=1.000 f1=1.000
bug tp=3 fp=0 fn=0 precision=1.000 recall=1.000 f1=1.000
account tp=1 fp=0 fn=1 precision=1.000 recall=0.500 f1=0.667
other tp=1 fp=1 fn=0 precision=0.500 recall=1.000 f1=0.667
macro F1 (plain average of the four): 0.833