A CI gate: threshold, baseline and critical cases
Prompt v3 has the same average as v2 but fails critical case t1. The gate blocks it. The exit status is printed instead of exiting.
A regression test for an LLM feature runs the eval on every prompt, model or code change and fails the build when quality drops below a threshold or a critical case breaks. Repeating cases and judging by majority keeps random noise from blocking good changes.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write gate_status(rate, threshold, failed_critical) returning 1 when the rate is below the threshold or the list failed_critical is not empty, and 0 otherwise. Print the status for (0.80, 0.70, []), (0.65, 0.70, []) and (0.90, 0.70, ["t1"]).
Write majority(passes, needed) where passes is a list of booleans from repeated runs of one case. Return True when at least needed are True. Print it for ([True, True, False, True, False], 3) and ([True, False, False, True, False], 3).
Explain it without notes
What should an LLM eval gate in CI check, and why is a single threshold not enough?
How do you handle flaky eval scores?
Which changes should trigger an eval run?
How do you keep eval cost and time under control in CI?
Expected output
prompt v2: pass rate 0.750 (baseline 0.625)
exit status: 0
prompt v3: pass rate 0.750 (baseline 0.625)
BLOCKED: critical case t1 failed
exit status: 1
prompt v1: pass rate 0.625 (baseline 0.625)
BLOCKED: pass rate 0.625 is below the threshold 0.700
exit status: 1