Validate the answer key and check coverage
Cheap checks on the dataset itself. They run in milliseconds and fail before a single model call is paid for.
A golden dataset is the answer key for your LLM feature: real inputs with labels a human agreed on. How you collect it, which edge cases you add, how big it is and how you keep it fresh decides whether your eval tells the truth.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write by_category(cases, results) that returns a dict from category to pass count, given the golden cases and a list of booleans. Print it for the v1 results [True, True, True, False, False, True, True, False] using the eight tickets' categories billing, bug, account, billing, bug, other, account, bug.
Write is_holdout(case_id) that uses tiny_hash(case_id) % 5 == 0, then list which of t1 to t10 are holdout. Use h = 7 and h = (h * 31 + ord(ch)) % 1000000007.
Explain it without notes
Where should the cases in a golden dataset come from, in order of preference?
Why do you split a dataset into dev and holdout, and why split by hash instead of at random?
Two people label 20 tickets and disagree on 6. What does that tell you and what do you do?
How large should a golden set be?
Expected output
golden set problems: 0
bad set problems:
duplicate id t8
t8: empty ticket
t8: unknown category biling
category counts: billing=2 bug=3 account=2 other=1
case kinds: typical=5 edge=3
warning: other has only 1 case(s), add more