A review queue: flagged requests plus a stable sample
Thumbs-down and fallback are simulated by the request number. The random sample is chosen by hashing the request id, so it is repeatable.
A golden set only covers the cases you thought of. Online evaluation learns from real traffic: user feedback, a stable sample of conversations for human review, and A/B tests that compare two prompts or models on live users while respecting statistical noise.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write margin(rate, n) returning 1.96 * sqrt(rate * (1 - rate) / n) rounded to 3 decimals. Print it for (0.6, 100) and (0.6, 400) and say how the second relates to the first.
Write clear_winner(rate_a, n_a, rate_b, n_b) that returns True when the gap between the rates is larger than sqrt(margin_a^2 + margin_b^2). Print it for (0.60, 100, 0.66, 100) and (0.60, 4000, 0.66, 4000).
Explain it without notes
Why do you need online evaluation when you already have a golden set?
How would you choose which production requests humans should review?
Why assign A/B variants by user id with a hash instead of randomly per request?
How do you decide that an A/B test has run long enough?
Expected output
req-7: thumbs down
req-11: fallback reply
req-14: thumbs down
req-15: random sample
req-21: thumbs down
req-22: fallback reply
req-25: random sample
req-28: thumbs down
req-33: fallback reply
req-35: thumbs down
req-42: thumbs down
req-44: fallback reply
req-45: random sample
req-49: thumbs down
req-55: fallback reply
req-56: thumbs down
60 requests, 16 queued for review: 13 flagged, 3 sampled