Topic 9.8
Evaluating in Production
In one line
A golden set is made of yesterday's questions. Real users bring new ones, so the product also needs signals from live use: thumbs with reasons, how often it refuses, which sources get opened, and a sample of real traffic that feeds the golden set.
Think of it like this
A restaurant keeps a test menu for the kitchen, but it also watches the dining room. Which dishes come back half eaten? Who leaves a comment card, and what does it say? The test menu tells you whether the cooking is consistent; the dining room tells you what real people actually order and think. Offline tests and live signals are two halves of one picture.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Online evaluation
- Measuring the product from live use, as opposed to a fixed test set.
- Thumbs reason
- The short reason a user picks when rating an answer down, such as 'wrong source'.
- No-answer rate
- The share of questions the product refused to answer.
- Implicit signal
- Behaviour that hints at quality without the user saying so, such as opening a source.
- Sampling
- Taking a small, fixed or random part of traffic for review.
- A/B test
- Sending different users to two versions and comparing a metric.
- Feedback loop
- Failures found in production become test rows, which prevent the same failure later.
Step by step
01Step 1: decide what to record per session
The minimum: the question (redacted if needed), whether the product answered, the thumbs and its reason, and which source the user opened. Add the run record from Topic 9.1 (ids retrieved, prompt size, answer) so a failure can be triaged later (Topic 9.6).
02Step 2: compute rates over the sessions
Every rate needs its denominator named. No-answer rate is over all sessions. Thumbs-down share is over votes, not sessions, because most people do not vote. Source-open rate is over answered sessions.
no_answer_rate = (total - len(answered)) / total
down_share = len(downs) / len(votes)
opened = sum(1 for e in answered if e[4] != "") / len(answered)03Step 3: count the reasons
Group the thumbs-down reasons. A pile of 'wrong source' reasons means retrieval; a pile of 'out of date' means freshness. Counts per reason turn feelings into work items.
04Step 4: pick candidates for the golden set
Every thumbs-down and every unanswered question that is not already in the set becomes a candidate, because those are the known failures. Skip questions already in the set.
05Step 5: sample the rest with a fixed seed
A fixed-seed sample of ordinary traffic keeps the golden set from becoming only failures. The seed makes the picks repeatable, so two people running the job see the same questions.
06Step 6: close the loop
A labelled candidate joins the golden set. The regression test (Topic 9.7) then protects the fix. The next time the same kind of question appears, the offline test already knows.
Try it yourself
- 1
Predict the rates
Count by hand: how many sessions have a vote, and how many of those are down? Do you get 0.700 and 0.571? Then run.
- 2
Change the denominator
Compute the thumbs-down share over all ten sessions instead of over votes. What number do you get, and which of the two better describes how users feel?
- 3
Another seed
Change the starting
stateto 7. Which two questions are sampled now? Is the sample random? Is it reproducible? - 4
Add a reason
Add a session with the thumbs-down reason
incomplete. How does the reasons line change? Which stage from Topic 9.6 would you look at first?
Code & diagrams
The ten sessions are made-up data. The sample uses the course's generator (state = state * 48271 % 2147483647, seed 42) and takes state % pool size, so all three languages pick the same two questions.
Expected output
sessions: 10
no-answer rate: 0.200
vote rate: 0.700
thumbs-down share: 0.571 of votes
opened a source: 0.625 of answered
down reasons: wrong source 2, no answer 1, out of date 1
add to the golden set: I forgot my login, how do I get back in? | Do you rent scooters? | Is there a student discount? | I scratched the bike, will I be charged? | Someone took my bike, what do I owe?
sampled for review: How far can I go on the electric one? | Can I get my money back if I cancel?Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Count people who did not vote as thumbs-down
In the Python tab, change the line that builds downs to downs = [e for e in EVENTS if e[2] != "up"], so everyone who did not give a thumbs up counts as a thumbs down. Run.
Myth vs fact
Myth
Thumbs-down share is the satisfaction rate.
Fact
Only a minority vote, and unhappy users vote more, so it overstates dissatisfaction. Use it as a trend, not as a score.
Myth
A lower no-answer rate means a better product.
Fact
It may mean the product is answering questions it should refuse. Check refusals against unanswerable golden rows.
Myth
Offline tests make live signals unnecessary.
Fact
Offline sets only contain questions you already thought of. Live traffic is where the new ones come from.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Make feedback easy but specific: two clicks (a thumb, then one reason) beat a free-text box that nobody fills in. Free text is useful to read, hard to count.
- ▸
Attach a version tag (prompt version, model, index build) to every session, so a change in a rate can be tied to a release (Topic 11.6).
- ▸
When you roll out a change to a small share of traffic first, watch the rates that move fast (no-answer rate, source opens) before the ones that need many votes.
- ▸
Review a weekly sample of sessions by hand, including good ones. Reading real conversations finds failure modes no metric was written for.
Remember this
- 1
Why it exists. A golden set is a snapshot. Users ask questions you did not predict, documents change, and the world drifts. A product with perfect offline numbers can still fail on next month's questions. Online evaluation finds those failures, and each one becomes a row in the golden set, which closes the loop.
- 2
Explicit signals. A thumbs up or down on each answer, with a short reason the user can pick when they press down: wrong source, out of date, incomplete, did not answer. Reasons matter more than the thumb: they point at a stage. 'Wrong source' suggests retrieval; 'out of date' suggests freshness (Topic 11.2); 'did not answer' on an answered question suggests the answer ignored the sources.
- 3
Implicit signals. What users do without saying anything: the no-answer rate (how often the product refused), whether they opened a cited source, whether they asked the same question again in different words, whether they escalated to a person. None is a verdict alone. A source click can mean 'I want to check' or 'this was not enough'. Treat them as trends to watch, and compare before and after a change.
- 4
The numbers in the example. Ten sessions: no-answer rate 0.200, vote rate 0.700, thumbs-down share 0.571 of the votes, and a source opened in 0.625 of the answered sessions. The down reasons: wrong source 2, no answer 1, out of date 1. With only ten sessions these are anecdotes, not statistics; the code is what matters, because it is the same code on ten million.
- 5
Biases to remember. Only some users vote, and unhappy users vote more than happy ones, so the thumbs-down share overstates real dissatisfaction. A low no-answer rate is not automatically good (it might mean the product is making things up), and a high one is not automatically bad (users may ask off-topic things). Read each signal against the others and against the offline numbers.
- 6
Turn traffic into golden rows. Every thumbs-down and every unanswered question is a candidate. Add a small fixed-seed sample of the rest so good traffic is represented too (the example draws two with the course's generator, so everyone gets the same picks). Then a person labels the right passages for each candidate before it enters the set; an unlabeled candidate is just a question. The example proposes five candidates and two sampled questions.
- 7
Privacy and cost. Questions can contain personal details. Redact them before storing or showing them to reviewers, keep them only as long as needed, and say what you log (Phase 10). If you compare two versions in an A/B test, you need enough sessions: with 200 sessions per version the standard error of a rate near 0.5 is about 0.035 (sqrt(0.25 / 200)), and the gap between two such rates has a standard error of about 0.05, so a 5-point gap is noise. Small products often learn faster from reading 20 failed sessions than from a statistical test.
Explain it without notes
Name three online signals and what each can and cannot tell you.
Why is the thumbs-down share computed over votes, and what bias remains?
How does production traffic improve the golden set?
Why can a very low no-answer rate be a warning sign?
Practice
Write rate(part, whole) returning the fraction as a 3-decimal string and raising a clear message when whole is 0. Print rate(4, 7).
Write top_reasons(reasons, n) that returns the n most common reasons, ties broken by first appearance. Print the top 2 for a list of five reasons.
Trade-offs
- ↔
Explicit feedback: clear intent, few responses. Implicit signals: plentiful, ambiguous.
- ↔
Reviewing sessions by hand: finds new failure modes, costs time and needs privacy care. Automated rates: scale, blind to new kinds of failure.
- ↔
A/B tests: causal evidence, need many sessions. Before-and-after comparisons: easy, confounded by anything else that changed.
Done when you can
Done when you can compute no-answer rate, vote rate, thumbs-down share and source-open rate with the right denominators.
Done when you can map each thumbs-down reason to a stage in the triage.
Done when you can turn live traffic into labelled golden-set rows.
Done when you can name two biases in thumbs data.