Split dev and holdout by hashing the case id
tiny_hash is a hand-written hash: h = (h * 31 + code) % 1000000007. The same in all three languages, so the output matches.
A golden dataset is the answer key for your LLM feature: real inputs with labels a human agreed on. How you collect it, which edge cases you add, how big it is and how you keep it fresh decides whether your eval tells the truth.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write by_category(cases, results) that returns a dict from category to pass count, given the golden cases and a list of booleans. Print it for the v1 results [True, True, True, False, False, True, True, False] using the eight tickets' categories billing, bug, account, billing, bug, other, account, bug.
Write is_holdout(case_id) that uses tiny_hash(case_id) % 5 == 0, then list which of t1 to t10 are holdout. Use h = 7 and h = (h * 31 + ord(ch)) % 1000000007.
Explain it without notes
Where should the cases in a golden dataset come from, in order of preference?
Why do you split a dataset into dev and holdout, and why split by hash instead of at random?
Two people label 20 tickets and disagree on 6. What does that tell you and what do you do?
How large should a golden set be?
Expected output
t1 hash=10372 -> dev
t2 hash=10373 -> dev
t3 hash=10374 -> holdout
t4 hash=10375 -> dev
t5 hash=10376 -> dev
t6 hash=10377 -> holdout
t7 hash=10378 -> dev
t8 hash=10379 -> dev
dev: 6 cases, v1 passes 3, v2 passes 5
holdout: 2 cases, v1 passes 2, v2 passes 1
holdout after adding t9..t12: t3, t6, t9, t12