Topic 9.7
Evaluating in Production
In one line
A golden set only covers the cases you thought of. Online evaluation learns from real traffic: user feedback, a stable sample of conversations for human review, and A/B tests that compare two prompts or models on live users while respecting statistical noise.
Think of it like this
A restaurant. The chef tastes every dish while developing the menu (the golden set), but only the customers show what really happens on a Friday night: which dishes come back half eaten, what people complain about, which new special sells. A good restaurant listens to the dining room as well as tasting in the kitchen. Online evaluation is listening to the dining room.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Online evaluation
- Measuring quality on real user traffic rather than on a fixed test set.
- Explicit feedback
- What users say directly, such as a thumbs-up or a rating.
- Implicit signal
- What users do that hints at quality, such as asking again or abandoning the page.
- Review queue
- A list of real conversations set aside for a person to read and judge.
- A/B test
- Showing two versions to different users at the same time and comparing the results.
- Margin of error
- How far an observed rate can plausibly be from the true rate because of luck.
- Guardrail metric
- A measure that must not get worse even while the main metric improves.
- Canary / gradual rollout
- Sending a small share of traffic to a new version first and growing it if it behaves.
Step by step
01Step 1: collect signals you can act on
Add a thumbs-up and thumbs-down to each answer and log them with the request id, the prompt version and the model version. Without the version in the log you cannot say which change caused a drop (Topic 11.2).
02Step 2: decide what goes to a human
The first example reviews every thumbs-down, every fallback reply, and a stable 10% sample of the remaining requests, chosen by bucket(request id, 100) < 10. Of 60 requests it queues a mix of flagged and sampled ones. Increase the sample when the feature is new, lower it when it is stable.
def bucket(text, n):
return tiny_hash(text) * 48271 % 2147483647 % n
in_sample = bucket("req-" + str(i), 100) < 1003Step 3: turn reviews into cases
A reviewer marks each queued item good or bad and, for bad ones, writes the right answer. Those become golden cases (Topic 9.2), so the offline eval gets harder and more realistic every month.
04Step 4: split users, not requests
The second example assigns each user to prompt A or B by bucket(user id, 2). The same user always sees the same prompt, which keeps their experience consistent. Assigning by request would let one person see both and blur the comparison.
05Step 5: compare with the noise in mind
For each prompt the code prints the thumbs-up rate and its margin of error. After 200 users the gap is smaller than the combined noise, so the verdict is to keep running. After 4000 users the gap is larger than the noise, so a decision is safe. The margin shrinks with the square root of the sample size: 4 times the users halves the margin.
06Step 6: roll out with guardrails
A winner is rolled out gradually with a switch back to the old version. Watch cost per request, latency and the fallback rate as guardrails during the rollout, not just the metric the test was about.
Try it yourself
- 1
Change the sample rate
In the first example change
SAMPLE_PERCENTto 25. How many requests are queued now? Does the request you already had in the queue stay in it? Why is that useful? - 2
Run longer
In the second example add
experiment(20000). What happens to the margins and to the verdict? Roughly how many users do you need to see a 3-point gap clearly? - 3
Make the gap smaller
Change B's chance from 66 to 62 and run again at 4000 users. Is the verdict still clear? What does this say about testing tiny improvements?
Code & diagrams
Thumbs-down and fallback are simulated by the request number. The random sample is chosen by hashing the request id, so it is repeatable.
Expected output
req-7: thumbs down
req-11: fallback reply
req-14: thumbs down
req-15: random sample
req-21: thumbs down
req-22: fallback reply
req-25: random sample
req-28: thumbs down
req-33: fallback reply
req-35: thumbs down
req-42: thumbs down
req-44: fallback reply
req-45: random sample
req-49: thumbs down
req-55: fallback reply
req-56: thumbs down
60 requests, 16 queued for review: 13 flagged, 3 sampledSimulated users with a true thumbs-up chance of 60% for A and 66% for B. A seeded generator makes the run repeatable. The margin is 1.96 * sqrt(rate * (1 - rate) / n).
Expected output
after 200 users:
prompt A: 68/101 thumbs up = 0.673 (+/- 0.091)
prompt B: 69/99 thumbs up = 0.697 (+/- 0.091)
gap 0.024, noise 0.129: not enough data, keep running
after 4000 users:
prompt A: 1197/2002 thumbs up = 0.598 (+/- 0.021)
prompt B: 1318/1998 thumbs up = 0.660 (+/- 0.021)
gap 0.062, noise 0.030: B is betterBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Declare a winner on the first look
In the second Python example, replace the line starting with verdict = by verdict = "B is better" if rate["B"] > rate["A"] else "A is better", so the noise is ignored.
Myth vs fact
Myth
Thumbs-up rates tell the whole story.
Fact
Few users click, and the ones who do are not typical. Combine feedback with sampled human review and with implicit signals, and watch cost and latency too.
Myth
If B is ahead after a day, ship B.
Fact
Early gaps are mostly noise. Without enough users, the leader changes from day to day. Decide the sample size up front.
Myth
Offline evals make online evaluation unnecessary.
Fact
Offline evals only know the cases you collected. Real traffic holds the surprises, and each one should become a new offline case.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Log the prompt version, model version, retrieval settings and a request id on every response. Without them a thumbs-down cannot be traced to the change that caused it.
- ▸
Mind the privacy of what you store: user text in review queues and logs is personal data. Redact before storing, restrict who can read it, and set a retention limit (Topics 10.5 and 11.2).
- ▸
Interleaving is a cheaper alternative to a full A/B test for ranked outputs, and shadow mode (run the new version on live traffic but show users the old answer) lets you compare a candidate safely with a judge (Topic 9.4) before any user sees it.
- ▸
When feedback is rare, a short in-product question ("Did this solve your problem?") on a sample of requests gives far more labels than waiting for thumbs.
Remember this
- 1
Why this exists. Offline evals use cases you chose. Real users send inputs you never imagined, in volumes and mixes you cannot predict, and the world changes: new products, new slang, new attacks. The only way to learn how the feature performs on real traffic is to measure it on real traffic. Online evaluation also feeds the golden set: every failure found in production becomes a new test case (Topic 9.2).
- 2
Explicit feedback is what users tell you: a thumbs up or down, a rating, a "report a problem" button, a retry or an edit. It is precise but rare (most users never click) and biased (angry users click more). Implicit signals are what users do: they copy the answer, ask the same question again, abandon the page, or hand over to a human agent. Both are noisy, so look at rates over many requests and at trends, not at one event.
- 3
Sampling for human review. You cannot read every conversation, so you build a review queue. Put in everything suspicious (thumbs-down, fallback replies, validation failures, very long or very expensive requests) and add a stable random sample of the rest, so you also see what no signal flags. Choose the sample by hashing the request id (the first example does), not by a fresh random draw, so the same request is always in or out and a reviewer can rebuild the queue. Remove personal data before humans see it (Topic 10.5).
- 4
A/B testing prompts and models. To learn whether prompt B really beats prompt A, show A to some users and B to others at the same time, and compare a measure you care about (thumbs-up rate, task completion, escalations, cost per task). Assign by a hash of the user id so that each user always sees the same variant. Run both at the same time, because traffic changes by day and hour.
- 5
Noise decides how long to run. Suppose true thumbs-up rates are 60% and 66%. With 200 users the observed rates move by several points just by luck, and the second example reports that the gap is smaller than the noise, so it says to keep running. With 4000 users the gap is larger than the noise. A rough rule of thumb is the margin 1.96 × sqrt(rate × (1 - rate) / n) on each rate. To combine the two margins, take the square root of the sum of their squares (the example does this). If the gap between two variants is smaller than that combined margin, you have not learned anything yet. Decide the sample size before you start and do not stop the moment one side is ahead.
- 6
Guardrail metrics. A variant can win on thumbs-up and lose on cost, latency, refusals or safety. Track a few guardrails alongside the main metric (cost per request, p95 latency, fallback rate, flagged-unsafe rate) and require that none gets worse beyond a limit. Roll out gradually (1%, 10%, 50%) and keep a switch to go back. Observability (Phase 11) gives you the data for all of this.
Explain it without notes
Why do you need online evaluation when you already have a golden set?
How would you choose which production requests humans should review?
Why assign A/B variants by user id with a hash instead of randomly per request?
How do you decide that an A/B test has run long enough?
Practice
Write margin(rate, n) returning 1.96 * sqrt(rate * (1 - rate) / n) rounded to 3 decimals. Print it for (0.6, 100) and (0.6, 400) and say how the second relates to the first.
Write clear_winner(rate_a, n_a, rate_b, n_b) that returns True when the gap between the rates is larger than sqrt(margin_a^2 + margin_b^2). Print it for (0.60, 100, 0.66, 100) and (0.60, 4000, 0.66, 4000).
Trade-offs
- ↔
Explicit feedback is clear but rare and biased; implicit signals are plentiful but ambiguous.
- ↔
A bigger review sample finds more problems but costs reviewer time and exposes more user data.
- ↔
Longer A/B tests give surer answers but delay improvements and keep half the users on the weaker variant.
Done when you can
Done when you can name explicit and implicit feedback signals for an LLM feature.
Done when you can build a review queue from flagged requests plus a stable hashed sample.
Done when you can assign A/B variants by hashing the user id.
Done when you can compute a margin of error and compare it with the gap between variants.
Done when you can turn a production failure into a golden case.