Topic 13.7
Reading Benchmarks Critically
In one line
A benchmark is a fixed test with a scoring rule, summarised as one number. The number can be inflated by test questions that leaked into training, hidden by noise on small tests, and irrelevant to your task, so use it to build a shortlist and your own evals to decide.
Think of it like this
An exam whose questions leaked a week before. One student scores top marks because they memorised the answers, not because they understand the subject. Another student, who understands better, scores lower. Looking only at the marks, you hire the wrong person. A benchmark score can be that kind of mark.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Benchmark
- A fixed set of test questions with a scoring rule, summarised as one number.
- Contamination
- Test questions that also appear in the training data, so a high score can mean memory, not skill.
- n-gram
- A run of n words in a row. Overlap of n-grams between two texts shows copied passages.
- Standard error
- A measure of how much a score would wobble if you ran a different sample of test items.
- Margin of error
- A range around a score or a gap. Differences smaller than it cannot be trusted.
- Saturation
- When top models all score near the maximum, so the benchmark can no longer separate them.
- Goodhart's law
- When a measure becomes a target, it stops being a good measure.
Step by step
01Step 1: ask what the number measures
Before comparing two numbers, find out the task, the number of items, how answers were scored and which prompt and settings were used. If you cannot find these, treat the number as marketing.
02Step 2: check for leaked test questions
The first example takes four test questions and a scrap of training text. Items 1 and 4 share most of their 5-word runs with the text and are flagged. Item 2 shares one run out of five, so it is below the 50 percent line but not above suspicion. Item 3 is clean.
def ngrams(text, n):
w = text.lower().split()
return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}
print(sorted(ngrams("a b c d", 3))) # ['a b c', 'b c d']03Step 3: recompute the score without the suspicious items
The model gets 3 of 4 on the full set, but only 1 of 2 on the items that look clean. With real tests you do this over hundreds of items, and a large gap between the two numbers is a warning sign.
04Step 4: compute the noise
The second example turns 41 and 43 correct out of 50 into percentages and standard errors, adds the two errors the way independent errors combine, and multiplies by 1.96 for a rough 95 percent margin. The gap (4 points) is far inside the margin (14 points), so the program says the results cannot rank the models.
05Step 5: size your own eval for the decision
If the gap you care about is 4 points, the last line of the example estimates how many items you need: about 644. If you can only afford 100 items, you can only detect larger gaps, or you can compare the models case by case on the same items (Topic 13.8), which reduces the noise because both are tested on the same questions.
06Step 6: use benchmarks for what they are good at
Shortlisting: which models are even in the right league for reasoning, code or long documents. Tracking: whether a new version of the same model improved on a stable test. Not for the final choice: that comes from your evals, your gates (Topic 13.1) and your costs (Topic 13.2).
Try it yourself
- 1
Change the n-gram size
In the first example set
N = 3. Which items become flagged, and why does a smaller n produce more false alarms on common phrases? - 2
Paraphrase the leak
Change the training text to say
which city is the capital of Australiainstead of the exact question. What does the overlap check say for item 1? What does this tell you about the limits of the check? - 3
Grow the test
In the second example set
n = 500and the counts to 410 and 430 (the same percentages). What happens to the margin? Does the verdict change?
Code & diagrams
The texts and the model's results are made up. Real checks run over millions of items with a hash set of n-grams.
Expected output
item 1: 5 of 5 5-grams in the training text (100 percent) -> LIKELY CONTAMINATED
item 2: 1 of 5 5-grams in the training text (20 percent) -> ok
item 3: 0 of 4 5-grams in the training text (0 percent) -> ok
item 4: 3 of 4 5-grams in the training text (75 percent) -> LIKELY CONTAMINATED
model accuracy on all items: 3 of 4
model accuracy on items that look clean: 1 of 2Counts are made up. The 1.96 margin is a rough normal approximation: it is a guide for the size of the noise, not a precise test.
Expected output
model A: 41 of 50 correct (82.0 percent), standard error 5.4 points
model B: 43 of 50 correct (86.0 percent), standard error 4.9 points
gap: 4 points, rough 95 percent margin: 14.3 points
verdict: the gap is inside the noise, so these results cannot rank the models
items needed before a 4-point gap clearly beats the noise: 644Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Use single words as n-grams
In the first example set N = 1.
Break #2
Call every positive gap real
In the second example change if diff_points < margin: to if diff_points < 0:.
Myth vs fact
Myth
The model with the highest benchmark score will be best on my task.
Fact
A benchmark measures its own tasks, prompts and scoring. Your task may need different skills, so test it.
Myth
A difference of a few points is a real difference.
Fact
On a test with tens or a few hundred items, differences of several points are often noise.
Myth
If I find no overlap, the benchmark is clean.
Fact
Paraphrased or translated copies slip past overlap checks, and you usually cannot see a provider's training data.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Prefer paired comparisons: run both models on the same items and look at the items where they differ. The noise of a paired difference is smaller than the noise of two independent averages, so you can detect smaller gaps with the same number of items.
- ▸
Keep your eval set private and out of prompts you send to services whose data policy allows training on inputs (Topic 13.1). A set that leaks into training stops being a test.
- ▸
Re-run your eval on every model change with the same prompt and settings. Also record the date and the exact model version, because a hosted model of the same name can change (Topic 3.8).
- ▸
Check score by slice (short inputs, long inputs, each language, each category), not only the average. A model that is better on average can be worse on the slice that matters most.
Remember this
- 1
Why this exists. Every model announcement comes with a table of benchmark numbers, and choosing by the biggest number is tempting because it is quick. To use these tables well you must know what the number measures, how it was produced, how much of the gap is just noise, and whether the model might have seen the questions. Then you decide how much weight the table deserves, which is usually: enough to build a shortlist, not enough to decide.
- 2
What a benchmark is. A fixed set of questions, a prompt format, and a scoring rule, collapsed into one number. Typical kinds are multiple-choice knowledge questions, maths word problems, programming tasks scored by running tests, and human votes comparing two answers. Each measures something narrow. A high score on programming puzzles says little about extracting fields from your invoices.
- 3
Contamination. If the test questions were in the training data, a model can score well by remembering them. Web-scale training data often contains public benchmark questions, quiz pages and discussion threads about them. A simple check is n-gram overlap: split each test item into runs of n words and count how many appear in the training text. The first example flags items with high overlap and shows the model's accuracy on all items versus only the items that look clean. A caveat: no overlap does not prove a clean test (a paraphrase slips through), and for hosted models you usually cannot inspect the training data at all.
- 4
Noise. A score from 50 test items is an estimate. If a model answers correctly with probability p, the standard error of its score is the square root of p x (1 - p) / n. The second example shows two models with 82 and 86 percent on 50 items: a four-point gap, with a rough margin of error of 14 points on the gap. The data cannot rank them, and it would take about 644 items before a four-point gap clearly stood out from the noise. Many published differences are smaller than the noise of the test.
- 5
Saturation, gaming and preference. When the best models all score near the top, the remaining differences say little. When a number becomes a target, models and prompts are tuned for it (the measure stops measuring what you wanted). Leaderboards built on human votes measure which answer people prefer, which can favour length, tone and confidence over correctness. Results also depend on the prompt, the number of examples shown, the settings and the scoring script, and self-reported numbers may not match those you get with your prompt.
- 6
Why your own evals win. They use your inputs, your format, your language and your failure costs, they are private so they cannot leak into training if you keep them out of public places, and you can size them for the decision at hand. A benchmark table is an opinion about other people's tasks. Your eval set (Phase 9) is evidence about yours. When the two disagree, trust yours, and check that yours is large enough to tell the candidates apart (the second example's last line).
Explain it without notes
What is benchmark contamination, and how can you check for it?
A model scores 86 percent and another 82 percent on a 50-item test. What can you conclude?
Why do your own evals beat public benchmarks for choosing a model?
How do you read a benchmark table critically?
Practice
Write ngram_overlap_percent(test, train, n) that returns the whole-number percentage of the test text's n-grams (sets of word runs, lower case, split on spaces) that appear in the training text. Print it for "a b c d" against "x a b c d y" with n = 3, and for "a b c d" against "a b x c d" with n = 3.
Write standard_error_points(correct, n) returning the standard error of a score in percentage points, rounded to one decimal. Print it for 41 correct out of 50.
Trade-offs
- ↔
Public benchmarks are free and comparable across models but may be contaminated, saturated or irrelevant to your task.
- ↔
Your own eval is relevant and private but costs effort to build and keep fresh, and a small one is noisy.
- ↔
A larger eval detects smaller gaps and costs more per run, in model calls and in human labelling.
Done when you can
Done when you can explain what a benchmark number does and does not tell you.
Done when you can run an n-gram overlap check and read its result.
Done when you can compute a standard error and a rough margin for a gap.
Done when you can estimate how many items you need to detect a gap you care about.
Done when you can list what to ask about a benchmark table before trusting it.