Five scorers and the replies that fool them
Each scorer is a few lines. The rows show what each one accepts and what it wrongly rejects or accepts.
A scorer turns one model reply into a pass or a number. Exact match, contains, regex, JSON validity, field accuracy, precision, recall, F1 and numeric tolerance cover most checks, and each is a few lines of code you can write yourself.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write prf(tp, fp, fn) returning precision, recall and F1, each rounded to 3 decimals, with 0.0 when a denominator is 0. Print the result for tp=3, fp=1, fn=2.
Write rel_close(want, got, tolerance) that returns True when got is a number within tolerance (as a fraction of want) of want, and False for anything that is not a number. Print the result for (100, "104", 0.05), (100, "106", 0.05) and (100, "abc", 0.05).
Explain it without notes
Define precision, recall and F1 for one class and say what each is good at revealing.
Why can accuracy be misleading, and what do you report instead?
When would you use exact match, contains, regex and JSON validity?
Why does the scorer itself need tests?
Expected output
exact_match pass ignores case and spaces
exact_match FAIL a full stop makes it fail
contains pass finds the key word
contains FAIL misses a paraphrase
regex_match pass right date shape
regex_match FAIL right date, wrong shape
json_valid pass clean JSON
json_valid FAIL JSON inside chatter
numeric_close pass within 1.0 of 120
numeric_close FAIL not a number at all
numeric_close FAIL 5 away from 120