Topic 9.2
A Golden Set for Retrieval
In one line
A golden set is a list of real questions with the passages that answer them, including questions the handbook cannot answer. Its quality caps the quality of every number you compute from it.
Think of it like this
A teacher writes the answer key before marking any exam. If the key has a wrong answer, every student who got it right is marked wrong, and nobody notices because everyone trusts the key. The golden set is the answer key for retrieval: each row says which passages should come back for a question.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Golden set
- A saved list of questions with the passages that should answer each.
- Unanswerable question
- A question the corpus cannot answer. The right product behaviour is to refuse.
- Relevant passage
- A passage that helps answer the question. There can be more than one.
- Coverage
- How much of the corpus and how many kinds of question the golden set exercises.
- Held-out set
- Rows you never tune on, kept to detect overfitting to the visible questions.
- Overfitting
- Improving the score on the test questions by tuning to them, without helping real users.
- Label
- The known right answer attached to a test row.
Step by step
01Step 1: choose the row shape
Each row needs an id, the question, a list of relevant passage ids and a kind. Plain data in a file or a table is enough: the evaluation code only reads it.
GOLDEN = [
("g1", "What does the monthly pass cost?", ["D8"], "keyword"),
("g2", "I forgot my login, how do I get back in?", ["D3"], "paraphrase"),
("g5", "Do you rent scooters?", [], "unanswerable"),
]02Step 2: collect questions the way users ask
Pull questions from support tickets, search logs and chat transcripts, and ask colleagues to write how they would ask. Keep typos, short forms and half sentences: users write them.
03Step 3: label by reading the passages
A person reads the corpus and writes the ids that answer each question. If two passages are needed, list both. If you are not sure, the row is ambiguous: reword the question or drop it.
04Step 4: validate the set with code
The first example finds three defects in a nine-row set: a duplicate question, an id (D99) that does not exist, and a row marked unanswerable that still lists a passage. It also lists the passages no question covers, which are blind spots.
05Step 5: compare model-written questions with real ones
The second example runs one retriever on both kinds of question for the same six passages. Questions copied from the passage score 1.000, questions written like users score 0.500. If you only had the first kind, you would believe the system is twice as good as it is.
06Step 6: try the same idea with real vectors
The lab below uses a real embedding model on an unanswerable question next to answerable ones. In a run of this lab the best score for the scooter question is about 0.28, while 'What does the monthly pass cost?' scores about 0.82 and the money-back question about 0.55. A threshold between roughly 0.30 and 0.50 would separate these cases, and Topic 8.4 explains how to pick one with data.
Try it yourself
- 1
Fix the planted defects
In the validator, repair g7 (remove it), g8 (use
D5) and g9 (give it an empty list). Run again. The problem count should reach 0, and the 'never asked about' line should still list passages: that is coverage work, not a defect. - 2
Write a user-style question
Add a seventh pair to
FROM_REAL_USERSfor passage D6, written the way a rider would say it, not copying the passage. Predict whether the word-overlap retriever finds D6 first, then run. - 3
Use the lab
Search the scooter question and note the best score. Then change the question to an answerable one such as
What does the monthly pass cost?and compare the best score. How far apart are the two numbers, and what would a threshold between them do to the borderline questions?
Code & diagrams
Three defects are planted on purpose: g7 repeats g3, g8 uses a passage that does not exist, and g9 is marked unanswerable but lists D5.
Expected output
questions: 9
keyword: 3
paraphrase: 3
two passages: 1
unanswerable: 2
problems (3):
g7: duplicate question
g8: unknown passage D99
g9: marked unanswerable but lists D5
passages never asked about: D2, D6, D7, D9, D10, D12The retriever is the word-overlap one from Phase 0: the passage that shares the most different words with the question wins, the first in handbook order on a tie.
Expected output
written from the text: hit rate at 1 = 1.000
found first: D1, D3, D5, D7, D9, D12
from real users: hit rate at 1 = 0.500
found first: D5, D9, D12Try: The handbook has no scooters. Press Search and read the best cosine score: that is what an unanswerable question looks like to a real embedding model. Then change the question to answerable ones (What does the monthly pass cost? Can I get my money back if I cancel?) and compare the best score each time. A refusal threshold has to sit between the two groups. Is there a clean gap?
Goals
- The Lost items passage scores 0.40 or less for the scooter question
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Stop checking that ids exist
In the Python tab of the validator, change if doc_id not in DOC_IDS: to if False:, so unknown passage ids are no longer reported. Run.
Myth vs fact
Myth
A model can write the whole golden set for me.
Fact
It can draft candidates quickly, but it copies the passage's wording, which makes keyword search look better than it is. Mix in real user questions and have a person check every label.
Myth
More rows always means a better set.
Fact
Duplicates and wrong labels make a big set worse than a small clean one. Add rows where the system is weak, and keep them correct.
Myth
Every question needs an answer.
Fact
A set with no unanswerable rows cannot tell a product that refuses properly from one that always makes something up.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Split the set by kind and report each part. An overall 0.80 can hide 1.00 on keyword questions and 0.50 on paraphrases, and the second number is the one users feel.
- ▸
Label with a second reviewer for the 20 most important rows and measure how often you disagree. Heavy disagreement means the questions are ambiguous, which no retriever can fix.
- ▸
When you rechunk, expected ids change. Store the source document and section with each label so you can regenerate passage ids, instead of relabelling by hand.
- ▸
Protect the set from leaking into training or tuning: if a re-ranker or prompt is tuned on the same questions you report, the reported number is optimistic. Keep a held-out slice.
Remember this
- 1
Why it exists. Metrics need something to compare against. Recall, MRR and nDCG (9.3 and 9.4) all start from the same fact per question: which passages are the right ones. The golden set stores that fact. It is the one piece of the evaluation you cannot compute, so it is the piece that decides whether the rest means anything.
- 2
What a row holds. A question, the ids of the passages that answer it (one or several), and a kind: keyword question, paraphrase, needs two passages, or unanswerable. Store ids, not text, so the row survives rewording a passage and the check needs no model. When you rechunk (Topic 4.4), ids change, so keep the passage's source document id as well, or re-map the rows.
- 3
Include questions the handbook cannot answer. If every row has an answer, the set rewards a product that always answers. A real product must also say 'I can't find that'. An unanswerable row has an empty list of passages, and the right behaviour is that nothing clears the evidence bar (Topic 8.4). Aim for roughly one in ten rows to be unanswerable, more if users often ask off-topic things.
- 4
Where questions come from. Best: real user questions from logs, support tickets and search queries. Next: colleagues asked to write how they would ask. Last: a model asked to write questions from each passage. Model-written questions are fast, but they copy the passage's own words, which flatters keyword search. The example runs the same word-overlap retriever on six questions written from the text and six written the way users talk: it finds the right passage first for all six of the first set and three of the second, hit rate 1.000 against 0.500. The real number is the lower one.
- 5
How big. Twenty good rows catch most regressions, and fifty to a few hundred let you see small differences. With only 10 rows one question moves a rate by 10 points, so small gains are noise. Cover every important document, every kind of question, and a few hard cases on purpose. Quality beats size: a duplicate row or a wrong id is worse than a missing row.
- 6
Check the set itself. The first example validates a golden set before it is trusted: duplicate questions, passages that do not exist, unanswerable rows that list a passage, answerable rows that list none, and passages no question ever asks about. A set with silent defects produces numbers that look precise and mean nothing.
- 7
Keep it fresh. Documents change and so do users. Add every user-reported failure as a row before fixing it (Topic 9.8 shows how to find candidates in production). Date the rows, review them when documents change, and never edit expected ids to make a metric go up. Keep a small held-out part you do not tune on, so you notice when you have overfitted to the visible questions.
Explain it without notes
What does a golden-set row contain, and why store passage ids instead of answer text?
Why include unanswerable questions?
Why are model-written questions biased, and how did the example show it?
How would you keep a golden set useful over a year?
Practice
Add a check to the validator that counts how many rows have more than one relevant passage and prints multi-passage rows: N. Run it on the nine-row set.
Write held_out(golden, every) that returns every every-th row (starting with the first) as a held-out list and the rest as the tuning list. Print the sizes for a list of 10 rows and every = 5.
Trade-offs
- ↔
Real user questions: realistic and slow to collect. Model-written questions: fast and biased towards the document's words. Mix both and label which is which.
- ↔
A bigger set: finer differences visible, more labelling and upkeep. A small clean set often beats a large messy one.
- ↔
Strict labels (only the best passage): simple metrics, but they punish a retriever that finds an equally good second passage. Graded labels (Topic 9.4) are fairer and cost more to write.
Done when you can
Done when you can write a golden-set row and say why it stores ids.
Done when you can list the defects a validator should catch.
Done when you can explain why unanswerable questions belong in the set.
Done when you can explain the bias of model-written questions using the 1.000 versus 0.500 result.
Done when you can describe how a user complaint becomes a new row.