Topic 9.7
Regression Tests in CI
In one line
A regression test runs the golden set on every change and fails the build when a metric falls below its floor or drops too far from the saved baseline. It turns 'we think it is better' into a check a machine enforces.
Think of it like this
A lift has an inspection sticker with a date and a weight limit. Nobody checks the cable each morning, but the sticker says it was checked against fixed rules, and the lift stops being used when the rules fail. A regression test is that inspection for your retrieval: fixed rules, run by a machine, with a pass or fail.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Regression
- A change that makes something worse that used to work.
- Baseline
- The saved metric values of the last accepted version.
- Floor (absolute threshold)
- A minimum value a metric must reach, whatever the baseline was.
- Maximum drop
- The most a metric may fall below the baseline before the test fails.
- CI (continuous integration)
- Automation that runs checks on every proposed change.
- Exit code
- The number a program returns: 0 means success, anything else means failure.
- Flaky test
- A test whose result changes between runs with no change to the code.
Step by step
01Step 1: freeze the inputs
Commit the golden set and the thresholds next to the code. Pin the corpus version and any seeds. A test that can silently change its inputs proves nothing.
02Step 2: save a baseline
Run the current accepted version once and store its metrics (and, for small sets, the place of the right passage per question). The example keeps the places directly: [1, 1, 2, 1, 3, 1, 2].
BASELINE = [1, 1, 2, 1, 3, 1, 2] # place of the right passage per question
CANDIDATE = [1, 1, 1, 1, 4, 2, 3] # after switching to smaller chunks03Step 3: compute the same metrics for the candidate
Run the candidate on the same questions and compute hit rate at 3 and MRR with the functions from Topics 9.3 and 9.4. The same code computes both sides, which keeps the comparison honest.
04Step 4: apply both kinds of threshold
For each metric check the floor and the drop. Print each check's own line so the failing one is obvious. The hit-rate line passes its floor (0.857 against 0.85) and fails its drop (0.143 against 0.05 allowed); MRR passes both.
05Step 5: print what changed
List every question whose place changed, marking worse ones and any that left the top k. A reviewer reads four lines instead of guessing from two averages.
06Step 6: wire it into CI
Add a job that runs the script on every pull request. In the YAML below, on: pull_request is the trigger, runs-on picks the machine, and each uses: line is a ready-made step (fetch the code, install Python). The last step runs your script, and the job fails when the script exits non-zero. Pin the action versions you use (check each action's releases page, because major versions move) and, if you want reviewers to open the report, add a step that uploads it as a build artifact.
Try it yourself
- 1
Make it pass
Change the candidate to
[1, 1, 1, 1, 3, 1, 2]and run. What are the three lines for each metric now? Which questions are listed as changed? - 2
Set a different drop
Raise
MAX_DROPto 0.15 and run the original candidate. Which check passes now? Is that the right way to make this candidate acceptable? - 3
A floor that bites
Raise
FLOOR_MRRto 0.75. Which metric fails its floor, and what does the drop check say about the same metric? - 4
Add a question
Add an eighth question with place 1 in both lists. How do the hit rate and MRR drops change, and why? (Remember: one question is now 1/8 of a rate.)
Code & diagrams
The 'places' are fixed numbers standing in for real retrieval runs, so all three languages print the same text. A real script would call sys.exit(1) (or process.exit(1), System.exit(1)) when the result is FAIL.
Expected output
hit rate at 3: baseline 1.000, candidate 0.857
floor 0.85: PASS
drop 0.143 (allowed 0.05): FAIL
MRR: baseline 0.762, candidate 0.726
floor 0.60: PASS
drop 0.036 (allowed 0.05): PASS
questions that changed:
login help: place 2 -> 1 (better)
late fee: place 3 -> 4 (worse, left the top 3)
flat tyre: place 1 -> 2 (worse)
battery range: place 2 -> 3 (worse)
result: FAIL (exit code 1)name: rag-eval
on: pull_request
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: python eval/run_eval.py --golden eval/golden.json --baseline eval/baseline.jsonBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Check only the floor
In the Python tab, change drop_ok = drop <= MAX_DROP to drop_ok = True, so only the floors are checked. Run.
Myth vs fact
Myth
If the overall number is above the floor, the change is safe.
Fact
A floor can sit far below where you were. The example loses 14 points of hit rate and still clears its floor.
Myth
A failing test should be fixed by lowering the threshold.
Fact
Lowering it removes the protection. If the trade is deliberate, update the baseline with a written reason; otherwise fix the change.
Myth
Evaluation belongs in a notebook.
Fact
A notebook is read when someone remembers. A CI job runs on every change and blocks the merge.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Separate fast and slow suites. Model-free retrieval metrics take seconds and can block every pull request. Judge-model runs take minutes and cost money: run them nightly and before releases, and compare with the last nightly result.
- ▸
Give the test a noise budget. If rerunning the same code can change a metric by 0.01, set the maximum drop above that, or you will train people to ignore red builds.
- ▸
Gate on the per-kind breakdown too (Topic 9.2): an overall hit rate can hold steady while paraphrase questions fall and keyword questions rise.
- ▸
Keep a list of 'must pass' rows (the questions behind past incidents). Failing any one of them blocks the merge regardless of averages.
Remember this
- 1
Why it exists. RAG has many levers that interact. A smaller chunk size can raise recall for short questions and lose long ones. A new embedding model can fix paraphrases and break product codes. Developers do not see these side effects by trying three questions. A regression is a change that makes something worse that used to work, and the only reliable defence is to run the same questions on every change.
- 2
What must stay fixed. The golden set (versioned with the code), the corpus snapshot or index build, the settings, and anything random. Retrieval with exact search is deterministic, so identical inputs give identical numbers. With approximate indexes or model calls, pin the seeds, set temperature to 0, and accept a small tolerance. If two runs of the same code differ, the test will fail for no reason and people will stop trusting it.
- 3
Two kinds of threshold. A floor is an absolute minimum: hit rate at 3 must stay at least 0.85, whatever happened before. A maximum drop compares with the saved baseline: no metric may fall more than 0.05 below it. The example needs both. The candidate (smaller chunks) scores hit rate 0.857, above the 0.85 floor, so the floor alone would let it through, yet it fell from 1.000, and the drop check fails it with a clear message.
- 4
Read the sizes of things. With 7 questions one question is 1/7 = 0.143 of the hit rate. A maximum drop of 0.05 is smaller than a single question, so in practice it means 'no question may leave the top 3'. That can be what you want, but decide it on purpose. On a small set, also list the individual questions that changed, as the example does: login help improved from place 2 to 1, late fee dropped from 3 to 4 and left the top 3, flat tyre fell from 1 to 2, battery range from 2 to 3. A person can judge those lines; an average hides them.
- 5
What triggers the test. Any change that can move retrieval or answers: the chunker, the embedding model or its version, index parameters, the retriever, the re-ranker, the prompt, the refusal threshold, the corpus. Run the fast, model-free retrieval checks on every pull request. Run the slower, costly ones (a judge model over many answers) nightly or before a release (see the CI/CD course, and LLM Engineering Topic 9.5).
- 6
Make the failure readable and the exit code honest. The script prints each metric, the floor, the drop and the questions that changed, then exits with code 1 on failure and 0 on success. A CI system treats a non-zero exit code as a failed step. The YAML below shows a minimal job; the exact steps depend on your CI system.
- 7
Changing the baseline is a decision, not a fix. When a change is a deliberate trade (a worse hit rate for a much better precision, say), update the saved baseline in the same pull request and explain why in the description, so a reviewer signs off on it. Never loosen a threshold just to make a red build green.
Explain it without notes
What is a regression test for RAG, and what must stay fixed for it to be reliable?
Explain the difference between a floor and a maximum drop with the example numbers.
Why list the individual questions that changed?
When should the baseline be updated?
Practice
Write gate(baseline, candidate, floor, max_drop) that returns a list of failure messages (empty when it passes) for one metric. Print the result for baseline=0.90, candidate=0.80, floor=0.70, max_drop=0.05.
Write changed(questions, before, after) that returns lines like late fee: 3 -> 4 for every question whose place changed. Print them for two questions.
Trade-offs
- ↔
Strict thresholds: catch small regressions, risk blocking harmless changes. Loose ones: fewer false alarms, more regressions slip through.
- ↔
Running everything on every pull request: safest, slowest and costliest. Splitting fast and slow suites: practical, with a delay on the slow signals.
- ↔
A big golden set: more stable numbers, slower runs and more labelling.
Done when you can
Done when you can list what must be fixed for a retrieval test to be repeatable.
Done when you can explain a floor and a maximum drop with numbers.
Done when you can write a script that exits non-zero on failure.
Done when you can say when it is right to update the baseline.