Contamination: n-gram overlap between test items and training text
The texts and the model's results are made up. Real checks run over millions of items with a hash set of n-grams.
A benchmark is a fixed test with a scoring rule, summarised as one number. The number can be inflated by test questions that leaked into training, hidden by noise on small tests, and irrelevant to your task, so use it to build a shortlist and your own evals to decide.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write ngram_overlap_percent(test, train, n) that returns the whole-number percentage of the test text's n-grams (sets of word runs, lower case, split on spaces) that appear in the training text. Print it for "a b c d" against "x a b c d y" with n = 3, and for "a b c d" against "a b x c d" with n = 3.
Write standard_error_points(correct, n) returning the standard error of a score in percentage points, rounded to one decimal. Print it for 41 correct out of 50.
Explain it without notes
What is benchmark contamination, and how can you check for it?
A model scores 86 percent and another 82 percent on a 50-item test. What can you conclude?
Why do your own evals beat public benchmarks for choosing a model?
How do you read a benchmark table critically?
Expected output
item 1: 5 of 5 5-grams in the training text (100 percent) -> LIKELY CONTAMINATED
item 2: 1 of 5 5-grams in the training text (20 percent) -> ok
item 3: 0 of 4 5-grams in the training text (0 percent) -> ok
item 4: 3 of 4 5-grams in the training text (75 percent) -> LIKELY CONTAMINATED
model accuracy on all items: 3 of 4
model accuracy on items that look clean: 1 of 2