Flaky scores: one run against five runs and a majority
Six simulated cases pass 70% to 95% of the time. The seeded generator makes the same 20 tries repeat in all three languages.
A regression test for an LLM feature runs the eval on every prompt, model or code change and fails the build when quality drops below a threshold or a critical case breaks. Repeating cases and judging by majority keeps random noise from blocking good changes.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write gate_status(rate, threshold, failed_critical) returning 1 when the rate is below the threshold or the list failed_critical is not empty, and 0 otherwise. Print the status for (0.80, 0.70, []), (0.65, 0.70, []) and (0.90, 0.70, ["t1"]).
Write majority(passes, needed) where passes is a list of booleans from repeated runs of one case. Return True when at least needed are True. Print it for ([True, True, False, True, False], 3) and ([True, False, False, True, False], 3).
Explain it without notes
What should an LLM eval gate in CI check, and why is a single threshold not enough?
How do you handle flaky eval scores?
Which changes should trigger an eval run?
How do you keep eval cost and time under control in CI?
Expected output
try 1: one run 4/6 -> BLOCKED, five runs 5/6 -> ok
try 2: one run 6/6 -> ok, five runs 6/6 -> ok
try 3: one run 4/6 -> BLOCKED, five runs 6/6 -> ok
try 4: one run 5/6 -> ok, five runs 6/6 -> ok
same unchanged prompt, 20 tries: one run blocked 5 times, five runs blocked 1 times