A regression gate with floors and a maximum drop
The 'places' are fixed numbers standing in for real retrieval runs, so all three languages print the same text. A real script would call sys.exit(1) (or process.exit(1), System.exit(1)) when the result is FAIL.
A regression test runs the golden set on every change and fails the build when a metric falls below its floor or drops too far from the saved baseline. It turns 'we think it is better' into a check a machine enforces.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write gate(baseline, candidate, floor, max_drop) that returns a list of failure messages (empty when it passes) for one metric. Print the result for baseline=0.90, candidate=0.80, floor=0.70, max_drop=0.05.
Write changed(questions, before, after) that returns lines like late fee: 3 -> 4 for every question whose place changed. Print them for two questions.
Explain it without notes
What is a regression test for RAG, and what must stay fixed for it to be reliable?
Explain the difference between a floor and a maximum drop with the example numbers.
Why list the individual questions that changed?
When should the baseline be updated?
Expected output
hit rate at 3: baseline 1.000, candidate 0.857
floor 0.85: PASS
drop 0.143 (allowed 0.05): FAIL
MRR: baseline 0.762, candidate 0.726
floor 0.60: PASS
drop 0.036 (allowed 0.05): PASS
questions that changed:
login help: place 2 -> 1 (better)
late fee: place 3 -> 4 (worse, left the top 3)
flat tyre: place 1 -> 2 (worse)
battery range: place 2 -> 3 (worse)
result: FAIL (exit code 1)