Topic 9.2
Golden Datasets
In one line
A golden dataset is the answer key for your LLM feature: real inputs with labels a human agreed on. How you collect it, which edge cases you add, how big it is and how you keep it fresh decides whether your eval tells the truth.
Think of it like this
A driving test route. The examiner does not pick random streets on the day. The test centre has a fixed route with a roundabout, a hill start, a narrow lane and a pedestrian crossing, because those are the places learners fail. A golden dataset is your feature's fixed route: ordinary cases plus the tricky spots where it is likely to fail.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Golden dataset
- A fixed set of inputs with trusted answers, used to score a feature.
- Label
- The correct answer a human assigned to a case.
- Edge case
- An unusual or tricky input where a feature is more likely to fail.
- Holdout set
- Cases you keep out of tuning so they can give an honest final score.
- Overfitting
- Tuning something so closely to the cases you can see that it does worse on new ones.
- Hash
- A function that turns text into a number; the same text always gives the same number.
- Inter-labeller agreement
- How often two people give the same label to the same case.
Step by step
01Step 1: collect from real life
Take tickets from the support inbox, remove names and account numbers, and keep the wording. Real tickets are shorter, messier and ruder than the ones you would write. Aim for every category and both priorities.
02Step 2: add the tricky ones
Add what you know is hard: a refund request that sounds like billing but is not urgent, a login problem that is really a bug, a thank-you message that contains no request. Mark each case typical or edge.
03Step 3: label with a guide
Write down what each label means in one line ("high = the customer cannot use the product or lost money right now"). Two people label independently, then compare. Change the guide, not just the label, when they disagree.
04Step 4: validate the key
A typo in a label (biling) makes a case impossible to pass and quietly lowers every score. The first example checks for duplicate ids, empty inputs and labels outside the allowed list, and counts cases per category so a thin category stands out (here other has one case).
if c["category"] not in CATEGORIES:
problems.append(c["id"] + ": unknown category " + c["category"])05Step 5: split by hash, not by hand
The second example sends each id through tiny_hash and puts ids whose hash divides by 3 into the holdout. Adding t9 to t12 later never moves an older case, so scores from last month stay comparable. A random shuffle would reshuffle everything each time.
06Step 6: keep it alive
Add a case for every production failure, retire cases that no longer reflect the product, and re-check the labels when the policy changes. A golden set that never changes is quietly out of date.
Try it yourself
- 1
Break the key on purpose
In the first example, change
"billing"in case t4 to"refund". Predict which problem line appears, then run it. - 2
Fill the thin category
Add an
othercase such as"Do you have a student discount?"(categoryother, prioritylow) toGOLDENand toKIND. Is the warning gone? - 3
Move the line
In the second example change
% 3to% 2. How many cases are in the holdout now? Which pass counts changed, and why does a smaller holdout make that number less trustworthy?
Code & diagrams
Cheap checks on the dataset itself. They run in milliseconds and fail before a single model call is paid for.
Expected output
golden set problems: 0
bad set problems:
duplicate id t8
t8: empty ticket
t8: unknown category biling
category counts: billing=2 bug=3 account=2 other=1
case kinds: typical=5 edge=3
warning: other has only 1 case(s), add moretiny_hash is a hand-written hash: h = (h * 31 + code) % 1000000007. The same in all three languages, so the output matches.
Expected output
t1 hash=10372 -> dev
t2 hash=10373 -> dev
t3 hash=10374 -> holdout
t4 hash=10375 -> dev
t5 hash=10376 -> dev
t6 hash=10377 -> holdout
t7 hash=10378 -> dev
t8 hash=10379 -> dev
dev: 6 cases, v1 passes 3, v2 passes 5
holdout: 2 cases, v1 passes 2, v2 passes 1
holdout after adding t9..t12: t3, t6, t9, t12Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A typo in the answer key
In the second Python example, change the category of case t3 in GOLDEN from "account" to "acount", and run it.
Myth vs fact
Myth
More cases always means a better eval.
Fact
Wrong labels, near-duplicates and cases that look like nothing real make a large set worse than a small careful one. Quality of cases first, then quantity.
Myth
I can ask the model to write my test cases.
Fact
A model can suggest ideas for edge cases, but cases it invents tend to look like what it already handles well. Real user inputs, with labels from a human, are what expose real failures.
Myth
Once labelled, the set never needs to change.
Fact
Products change, policies change, and users find new ways to fail. A good set gains a case with every bug and loses cases that no longer apply.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep dataset files as plain text in the repo (JSON lines or a typed list), reviewed in pull requests like code. A label change then has an author and a reason.
- ▸
Report scores by slice (by category, by
edgevstypical, by input length or language). A good average with a bad slice is the usual shape of a real production problem. - ▸
Beware leakage: if tickets in the set also appear in your few-shot examples (Topic 3.3), the model has seen the answers and the score is too high. Keep examples and test cases disjoint.
- ▸
Estimate the noise. If you run the same prompt twice and the score moves by 0.05, you cannot trust a difference of 0.03 between two prompts. Topic 9.5 deals with flaky scores.
Remember this
- 1
Why this exists. You cannot measure quality without something to measure against. Eval scores are only as good as the cases and labels behind them. A set built from made-up, friendly inputs gives a high score and a false sense of safety. A set of real inputs with correct labels gives a score you can trust.
- 2
Where cases come from. Best first: real inputs from your users or logs (with personal data removed, Topic 10.5). Next: examples from the people who will use the feature (support agents know the nasty tickets). Last: invented cases, for situations you expect but have not seen yet. Mark where each case came from (
realorinvented) so you can see how much of the set is real. - 3
Include edge cases on purpose. A ticket that mentions both a login and an error code. A message with no problem in it at all. An empty message. Text in another language. A reply that the model tends to wrap in chatter. Edge cases are rare in real traffic but are where regressions hide. Label each case
typicaloredgeso reports can show both. - 4
Labelling is the hard part. Two people looking at the same ticket sometimes disagree (is a refund request low or high priority?). Write a short labelling guide, have two people label a sample, and look at the cases where they differ: those are either unclear rules or genuinely ambiguous cases. Fix the rule, or drop or mark the ambiguous case. Disagreement between labellers is also a ceiling: a model cannot be expected to beat the humans.
- 5
How big? Start with 20 to 50 cases, as long as they cover every category and priority. With only 8 cases (as in our example), one case is 12.5% of the score, so small differences are noise. The more cases you have, the smaller a difference you can trust. Grow the set from failures: each production bug becomes a new case.
- 6
Keep a holdout. If you tune your prompt while looking at all cases, the prompt starts to fit those cases, just like a student who memorises last year's exam. Keep part of the set (the holdout) that you only look at when you decide whether to ship. A stable way to split is to hash each case's id: the same id always goes to the same side, even when you add cases later (the second example).
Explain it without notes
Where should the cases in a golden dataset come from, in order of preference?
Why do you split a dataset into dev and holdout, and why split by hash instead of at random?
Two people label 20 tickets and disagree on 6. What does that tell you and what do you do?
How large should a golden set be?
Practice
Write by_category(cases, results) that returns a dict from category to pass count, given the golden cases and a list of booleans. Print it for the v1 results [True, True, True, False, False, True, True, False] using the eight tickets' categories billing, bug, account, billing, bug, other, account, bug.
Write is_holdout(case_id) that uses tiny_hash(case_id) % 5 == 0, then list which of t1 to t10 are holdout. Use h = 7 and h = (h * 31 + ord(ch)) % 1000000007.
Trade-offs
- ↔
A larger set is steadier but costs more to label and run; a small set is fast but noisy.
- ↔
Real cases are realistic but contain personal data you must clean; invented cases are safe but optimistic.
- ↔
A holdout gives an honest score but means some labelled cases never guide tuning.
Done when you can
Done when you can name three sources of golden cases and which is best.
Done when you can add typical and edge cases and tag them.
Done when you can write a dataset validator that catches typos, duplicates and thin categories.
Done when you can split dev and holdout by hashing ids and explain why that is stable.
Done when you can say how a production failure becomes a new case.