Validate a golden set before trusting it
Three defects are planted on purpose: g7 repeats g3, g8 uses a passage that does not exist, and g9 is marked unanswerable but lists D5.
A golden set is a list of real questions with the passages that answer them, including questions the handbook cannot answer. Its quality caps the quality of every number you compute from it.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Add a check to the validator that counts how many rows have more than one relevant passage and prints multi-passage rows: N. Run it on the nine-row set.
Write held_out(golden, every) that returns every every-th row (starting with the first) as a held-out list and the rest as the tuning list. Print the sizes for a list of 10 rows and every = 5.
Explain it without notes
What does a golden-set row contain, and why store passage ids instead of answer text?
Why include unanswerable questions?
Why are model-written questions biased, and how did the example show it?
How would you keep a golden set useful over a year?
Expected output
questions: 9
keyword: 3
paraphrase: 3
two passages: 1
unanswerable: 2
problems (3):
g7: duplicate question
g8: unknown passage D99
g9: marked unanswerable but lists D5
passages never asked about: D2, D6, D7, D9, D10, D12