Is the gap bigger than the noise?
Counts are made up. The 1.96 margin is a rough normal approximation: it is a guide for the size of the noise, not a precise test.
A benchmark is a fixed test with a scoring rule, summarised as one number. The number can be inflated by test questions that leaked into training, hidden by noise on small tests, and irrelevant to your task, so use it to build a shortlist and your own evals to decide.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write ngram_overlap_percent(test, train, n) that returns the whole-number percentage of the test text's n-grams (sets of word runs, lower case, split on spaces) that appear in the training text. Print it for "a b c d" against "x a b c d y" with n = 3, and for "a b c d" against "a b x c d" with n = 3.
Write standard_error_points(correct, n) returning the standard error of a score in percentage points, rounded to one decimal. Print it for 41 correct out of 50.
Explain it without notes
What is benchmark contamination, and how can you check for it?
A model scores 86 percent and another 82 percent on a 50-item test. What can you conclude?
Why do your own evals beat public benchmarks for choosing a model?
How do you read a benchmark table critically?
Expected output
model A: 41 of 50 correct (82.0 percent), standard error 5.4 points
model B: 43 of 50 correct (86.0 percent), standard error 4.9 points
gap: 4 points, rough 95 percent margin: 14.3 points
verdict: the gap is inside the noise, so these results cannot rank the models
items needed before a 4-point gap clearly beats the noise: 644