An A/B test that knows when it has seen enough
Simulated users with a true thumbs-up chance of 60% for A and 66% for B. A seeded generator makes the run repeatable. The margin is 1.96 * sqrt(rate * (1 - rate) / n).
A golden set only covers the cases you thought of. Online evaluation learns from real traffic: user feedback, a stable sample of conversations for human review, and A/B tests that compare two prompts or models on live users while respecting statistical noise.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write margin(rate, n) returning 1.96 * sqrt(rate * (1 - rate) / n) rounded to 3 decimals. Print it for (0.6, 100) and (0.6, 400) and say how the second relates to the first.
Write clear_winner(rate_a, n_a, rate_b, n_b) that returns True when the gap between the rates is larger than sqrt(margin_a^2 + margin_b^2). Print it for (0.60, 100, 0.66, 100) and (0.60, 4000, 0.66, 4000).
Explain it without notes
Why do you need online evaluation when you already have a golden set?
How would you choose which production requests humans should review?
Why assign A/B variants by user id with a hash instead of randomly per request?
How do you decide that an A/B test has run long enough?
Expected output
after 200 users:
prompt A: 68/101 thumbs up = 0.673 (+/- 0.091)
prompt B: 69/99 thumbs up = 0.697 (+/- 0.091)
gap 0.024, noise 0.129: not enough data, keep running
after 4000 users:
prompt A: 1197/2002 thumbs up = 0.598 (+/- 0.021)
prompt B: 1318/1998 thumbs up = 0.660 (+/- 0.021)
gap 0.062, noise 0.030: B is better