Questions copied from the text flatter keyword search
The retriever is the word-overlap one from Phase 0: the passage that shares the most different words with the question wins, the first in handbook order on a tie.
A golden set is a list of real questions with the passages that answer them, including questions the handbook cannot answer. Its quality caps the quality of every number you compute from it.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Add a check to the validator that counts how many rows have more than one relevant passage and prints multi-passage rows: N. Run it on the nine-row set.
Write held_out(golden, every) that returns every every-th row (starting with the first) as a held-out list and the rest as the tuning list. Print the sizes for a list of 10 rows and every = 5.
Explain it without notes
What does a golden-set row contain, and why store passage ids instead of answer text?
Why include unanswerable questions?
Why are model-written questions biased, and how did the example show it?
How would you keep a golden set useful over a year?
Expected output
written from the text: hit rate at 1 = 1.000
found first: D1, D3, D5, D7, D9, D12
from real users: hit rate at 1 = 0.500
found first: D5, D9, D12