PHASE 9Advanced ~25 min· topic 3 of 8

Topic 9.3

Scoring Functions

In one line

A scorer turns one model reply into a pass or a number. Exact match, contains, regex, JSON validity, field accuracy, precision, recall, F1 and numeric tolerance cover most checks, and each is a few lines of code you can write yourself.

Think of it like this

Marking different school exercises. A spelling test is exact: the word is right or wrong. A short answer is marked with a checklist: did it mention the key idea? A maths answer is right if the number is right, even if the working is written differently. Each kind of question needs its own way of marking, and picking the wrong one gives unfair marks.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Scorer: A function that turns a model reply and a wanted answer into pass/fail or a number.
Exact match: The reply equals the wanted answer, usually after trimming spaces and ignoring case.
Regex (regular expression): A pattern that describes the shape of a piece of text, such as four digits, a dash, two digits.
True positive, false positive, false negative: Right yes, wrong yes, and missed yes for one class.
Precision: Of the answers the model called this class, the share that really are.
Recall: Of the cases that really are this class, the share the model found.
F1: One number from precision and recall that is high only when both are.
Macro average: The plain average over classes, so a small class counts as much as a big one.

Step by step

01Step 1: start with the cheapest scorer that works

For a label, use exact match. For a reply that must contain a fact, use contains. For a format, use regex or a parser. Only when none of these can decide do you reach for a judge model. The first example runs five scorers on eleven small replies, including the ones that fool them.

02Step 2: know how each scorer is fooled

exact_match fails on a full stop. contains misses a paraphrase and accepts a negation. regex_match accepts a shape that is not a real date. json_valid rejects JSON hidden inside chatter, which is right if your code needs plain JSON. numeric_close must treat "about 120" as a fail rather than crash.

scorers.pywhole filepython

def numeric_close(want, got, tolerance):
    try:
        return abs(float(got) - want) <= tolerance
    except ValueError:
        return False

03Step 3: score each field, not only the whole

The second example parses prompt v2's replies and counts, for each field, how many are right. Category and priority are each right 7 times in 8, but the whole object is right only 6 times, because the two mistakes sit in different cases.

04Step 4: count tp, fp and fn for every class

For account: t7 is a true positive. t3 is a false negative, because it is an account ticket that the model called other. For other: t6 is a true positive, and t3 is a false positive, the same mistake seen from the other side. So account has high precision (1.000) and low recall (0.500), and other is the reverse.

Step 4: count tp, fp and fn for every classdiagram

Rendering diagram…

05Step 5: combine into F1 and macro F1

F1 for account is 2 × 1.000 × 0.500 / 1.500 = 0.667. Same for other. billing and bug are perfect (1.000). The plain average is (1 + 1 + 0.667 + 0.667) / 4 = 0.833. Accuracy for category was 0.875: the macro F1 is lower because it counts the small classes as much as the big ones.

06Step 6: put the scorers in a registry

In the harness (Topic 9.8) each case carries the name of its scorer, so structured cases use field scorers and free-text cases use contains or a judge. Test scorers like any other code: they decide whether a change ships.

Try it yourself

1
Fool the scorer
Add a row to the first example: contains("refund", "We cannot refund this"). Predict the result, then run. What would a better check look like for answers that must grant the refund?
2
Work one class by hand
In the second example, take other. Write down tp, fp and fn on paper from the table of replies, compute precision, recall and F1 yourself, then compare with the printed line.
3
Change one reply
Change the reply for t2 under prompt v2 from bug to account. Predict which two classes change and in which direction (precision or recall), then run.

Code & diagrams

Five scorers and the replies that fool them New tab

Each scorer is a few lines. The rows show what each one accepts and what it wrongly rejects or accepts.

Expected output

exact_match   pass  ignores case and spaces
exact_match   FAIL  a full stop makes it fail
contains      pass  finds the key word
contains      FAIL  misses a paraphrase
regex_match   pass  right date shape
regex_match   FAIL  right date, wrong shape
json_valid    pass  clean JSON
json_valid    FAIL  JSON inside chatter
numeric_close pass  within 1.0 of 120
numeric_close FAIL  not a number at all
numeric_close FAIL  5 away from 120

Per-field accuracy, precision, recall and F1 by hand New tab

Scores prompt v2's replies on the golden set. Every number can be checked with a pencil: tp, fp and fn are counted per class.

Expected output

category accuracy: 7/8 = 0.875
priority accuracy: 7/8 = 0.875
whole object right: 6/8 = 0.750
billing  tp=2 fp=0 fn=0 precision=1.000 recall=1.000 f1=1.000
bug      tp=3 fp=0 fn=0 precision=1.000 recall=1.000 f1=1.000
account  tp=1 fp=0 fn=1 precision=1.000 recall=0.500 f1=0.667
other    tp=1 fp=1 fn=0 precision=0.500 recall=1.000 f1=0.667
macro F1 (plain average of the four): 0.833

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

A scorer that is too strict

In the first Python example, change the body of exact_match to return got == want.

terminal

$ Run

── what you'll see ──

exact_match FAIL ignores case and spaces

exact_match FAIL a full stop makes it fail

contains pass finds the key word

contains FAIL misses a paraphrase

regex_match pass right date shape

regex_match FAIL right date, wrong shape

json_valid pass clean JSON

json_valid FAIL JSON inside chatter

numeric_close pass within 1.0 of 120

numeric_close FAIL not a number at all

numeric_close FAIL 5 away from 120

Myth vs fact

Myth

Accuracy is enough.

Fact

With uneven classes accuracy hides failures: always guessing the common class scores high. Look at precision and recall per class.

Myth

A higher F1 always means better.

Fact

F1 weights precision and recall equally. If a miss is much worse than a false alarm (or the reverse), report both numbers and set the threshold for the one you care about.

Myth

Contains is a good scorer for free-text answers.

Fact

It accepts negations and wrong-but-related text and rejects correct paraphrases. It is fine for a required keyword, not for judging meaning.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Prefer scorers that return a number between 0 and 1 over a boolean when you can (field accuracy, similarity). You can then set a threshold and also see how far a case is from passing.
▸
Always count a reply that fails to parse as a miss in the denominator. Silently dropping unparseable replies makes the score look better the worse the format gets.
▸
Macro, micro and weighted averages answer different questions. Macro treats each class equally, micro pools all decisions (for single-label classification it equals accuracy), weighted follows class sizes. Say which one a number is.
▸
For text similarity there are metrics such as ROUGE and BLEU (counts of overlapping words) and embedding similarity (Topic 1.3). They are cheap, but correlate loosely with human judgment on open-ended answers; use them to flag change, not to prove quality.

Remember this

1
Why this exists. The model's reply is free text, but an eval needs a number. A scorer is a function from (the wanted answer, the model's reply) to pass/fail or a score from 0 to 1. Cheap, rule-based scorers are fast, free, repeatable and never tired, so use them wherever a rule can decide. Use a model as judge (Topic 9.4) only for what rules cannot score.
2
Exact match compares after small clean-ups (trim spaces, ignore case). Use it for labels such as billing. It is strict: billing. fails. Contains checks that a key phrase appears. It is looser and also easier to fool: a reply that says "we will not refund you" contains refund. Regex (a pattern language for text shapes) checks a format such as 2026-10-11; it checks the shape, not whether the date is real.
3
JSON validity is the first scorer for any feature that must return JSON: does the whole reply parse? A reply wrapped in chatter fails, because the code that reads it would fail too (Topic 4.1). After validity, check keys and values (Topics 4.4 and 4.5), then score each field on its own. Per-field accuracy tells you where the feature is weak: category 0.875 and priority 0.875 are different problems from whole-object 0.750.
4
Precision, recall and F1 are for classification, and they matter when classes are uneven. Take one class, say account. True positives (tp): tickets that are account and were called account. False positives (fp): called account but are not. False negatives (fn): are account but were called something else. Precision = tp / (tp + fp): when the model says account, how often is it right? Recall = tp / (tp + fn): of the real account tickets, how many did it find? F1 is their harmonic mean, 2 × precision × recall / (precision + recall): it is high only when both are high.
5
Why accuracy alone can mislead. If 95 of 100 tickets are other, a model that always answers other is 95% accurate and useless. Per-class recall shows it finds none of the bugs. Averaging F1 over classes gives the macro F1 (every class counts equally). Which of precision or recall you care about depends on the cost: for a fraud alert, missing fraud (low recall) is worse; for an automatic ban, wrongly banning (low precision) is worse.
6
Numeric tolerance. When the answer is a number (a total, a price), compare with a tolerance: abs(got - want) <= tolerance, or a relative tolerance for big numbers. Parse safely: "about 120" is not a number and should fail, not crash the eval. Whatever scorer you pick, test the scorer itself on a few known-good and known-bad replies (the first example does).

Explain it without notes

Define precision, recall and F1 for one class and say what each is good at revealing.

Why can accuracy be misleading, and what do you report instead?

When would you use exact match, contains, regex and JSON validity?

Why does the scorer itself need tests?

Practice

Write prf(tp, fp, fn) returning precision, recall and F1, each rounded to 3 decimals, with 0.0 when a denominator is 0. Print the result for tp=3, fp=1, fn=2.

Write rel_close(want, got, tolerance) that returns True when got is a number within tolerance (as a fraction of want) of want, and False for anything that is not a number. Print the result for (100, "104", 0.05), (100, "106", 0.05) and (100, "abc", 0.05).

Trade-offs

↔
Strict scorers are repeatable but reject right answers worded differently; loose scorers accept wrong answers.
↔
Rule-based scorers are free and fast but only cover what a rule can state; judges cover meaning but cost money and have their own errors.
↔
Many metrics give a fuller picture but make a release decision harder; pick one headline number and keep the rest for diagnosis.

Done when you can

Done when you can write exact match, contains, regex, JSON validity and numeric tolerance scorers.
Done when you can count tp, fp and fn for a class from a table of predictions.
Done when you can compute precision, recall, F1 and macro F1 by hand.
Done when you can explain how each scorer can be fooled.
Done when you can choose precision or recall to prioritise from the cost of mistakes.

Command Palette

Words you'll meet

Step by step

01Step 1: start with the cheapest scorer that works

02Step 2: know how each scorer is fooled

03Step 3: score each field, not only the whole

04Step 4: count tp, fp and fn for every class

05Step 5: combine into F1 and macro F1

06Step 6: put the scorers in a registry

Try it yourself

Fool the scorer

Work one class by hand

Change one reply

Code & diagrams

Break it on purpose

A scorer that is too strict

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can