PHASE 9Advanced ~26 min· topic 5 of 8

Topic 9.5

Regression Tests in CI

In one line

A regression test for an LLM feature runs the eval on every prompt, model or code change and fails the build when quality drops below a threshold or a critical case breaks. Repeating cases and judging by majority keeps random noise from blocking good changes.

Think of it like this

A smoke alarm. You do not check the kitchen for fire every ten minutes; a small device watches all the time and screams the moment something crosses a line. A CI gate is that alarm for quality: it watches every change to the prompts, the model name and the code, and raises the alarm before a regression reaches users.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

CI (continuous integration): A server that automatically runs checks (build, tests, evals) on every change to the code.
Gate: A check that must pass before a change can be merged or released.
Exit status: The number a program ends with: 0 means success, anything else means failure. CI reads it.
Threshold: The minimum score a change must reach.
Baseline: The score of the version currently in production, used for comparison.
Critical case: A case that must never fail, whatever the average score is.
Flaky: A check whose result changes between runs although nothing changed.
CI artifact: A file a CI job saves for you to download later, such as the full eval report.

Step by step

01Step 1: turn the eval into a function that returns problems

gate(system) runs the eval and returns the pass rate and a list of problems. An empty list means the change may go ahead. Keeping the result as data makes the gate easy to test and easy to print.

02Step 2: add the three rules

The rate must be at least THRESHOLD. It must not fall more than MAX_DROP below the baseline. And none of the critical cases (t1 and t2) may fail. Each broken rule adds one readable line to the problems list.

gate.pywhole filepython

if rate < THRESHOLD:
    problems.append(f"pass rate {rate:.3f} is below the threshold {THRESHOLD:.3f}")
if case["id"] in CRITICAL and not ok:
    problems.append("critical case " + case["id"] + " failed")

03Step 3: exit with a status

In a real job the script ends with sys.exit(1) when there are problems and 0 otherwise, and CI turns the check red or green. The examples print the status instead of exiting, so they can run in a page. Prompt v2 gets 0, v3 gets 1 (t1 fails although the average is the same 0.750), v1 gets 1 (below the threshold).

Step 3: exit with a statusdiagram

Rendering diagram…

04Step 4: see the flakiness

The second example simulates six cases that each pass 70% to 95% of the time. With one run per try, an unchanged prompt can fail the 5-of-6 gate by luck. The code repeats the same 20 tries with five runs per case and a majority vote per case.

05Step 5: wire it into a pipeline

The YAML block shows a GitHub Actions workflow with two jobs: a replay job on every pull request that touches prompts (no key, no cost) and a nightly live job. The pipeline concepts (triggers, jobs, secrets, artifacts) are in the CI/CD course at /cicd. The test-framework block shows the same gate written as an ordinary test in pytest, JUnit and Vitest, so the CI system needs no special support.

06Step 6: tune the gate by watching it

After a few weeks, look at how often the gate blocked changes that were fine (too strict or too flaky) and how often a bad change got through (too loose). Move the threshold and the repeat count accordingly. A gate people trust gets used; a gate people bypass is worse than none.

Try it yourself

1
Predict the exit status
Before running the first example, write down the status for v2, v3 and v1. Then run it. Which rule blocked each?
2
Move the threshold
Set THRESHOLD to 0.800. Which prompts are blocked now? Is v2 (the best prompt you have) now unshippable, and what would you do about that?
3
More repeats
In the second example call majority_score(9) and change the vote test from >= 3 to >= 5. How does the number of blocked tries change? What do repeats cost in model calls?

Code & diagrams

A CI gate: threshold, baseline and critical cases New tab

Prompt v3 has the same average as v2 but fails critical case t1. The gate blocks it. The exit status is printed instead of exiting.

Expected output

prompt v2: pass rate 0.750 (baseline 0.625)
  exit status: 0
prompt v3: pass rate 0.750 (baseline 0.625)
  BLOCKED: critical case t1 failed
  exit status: 1
prompt v1: pass rate 0.625 (baseline 0.625)
  BLOCKED: pass rate 0.625 is below the threshold 0.700
  exit status: 1

Flaky scores: one run against five runs and a majority New tab

Six simulated cases pass 70% to 95% of the time. The seeded generator makes the same 20 tries repeat in all three languages.

Expected output

try 1: one run 4/6 -> BLOCKED, five runs 5/6 -> ok
try 2: one run 6/6 -> ok, five runs 6/6 -> ok
try 3: one run 4/6 -> BLOCKED, five runs 6/6 -> ok
try 4: one run 5/6 -> ok, five runs 6/6 -> ok
same unchanged prompt, 20 tries: one run blocked 5 times, five runs blocked 1 times

The gate as an ordinary test: pytest, JUnit 5 and Vitestpython

Not run here: it needs your own project. The idea is the same in each: call your eval, compute the rate, assert it is above the threshold.

# tests/test_triage_eval.py  (run with: pytest)
from triage_eval import run_eval  # your harness, built in Topic 9.8

THRESHOLD = 0.70


def test_pass_rate_stays_above_threshold():
    results = run_eval("v2")  # a list of booleans, one per golden case
    rate = sum(results) / len(results)
    assert rate >= THRESHOLD, f"pass rate {rate:.3f} is below {THRESHOLD}"


def test_critical_cases_never_fail():
    results = run_eval("v2")
    assert results[0] and results[1], "t1 and t2 must always pass"

llm-eval.ymlyaml

A GitHub Actions workflow: replayed responses on every pull request, the live model nightly. Check the action versions in the Actions marketplace when you copy it.

# .github/workflows/llm-eval.yml
name: llm-eval
on:
  pull_request:
    paths: ["prompts/**", "evals/**", "src/llm/**"]
  schedule:
    - cron: "0 3 * * *"   # nightly, against the live model

jobs:
  replay:   # every pull request: recorded responses, no key, no cost
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python evals/run_eval.py --mode replay --threshold 0.70   # a non-zero exit fails the check

  live:     # nightly: the real model, repeated runs
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python evals/run_eval.py --mode live --repeat 5 --threshold 0.70
        env:
          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Gate on the average only

In the first Python example, change the first line of rules to CRITICAL = [], so no case is critical.

terminal

$ Run

── what you'll see ──

prompt v2: pass rate 0.750 (baseline 0.625)

exit status: 0

prompt v3: pass rate 0.750 (baseline 0.625)

exit status: 0

prompt v1: pass rate 0.625 (baseline 0.625)

BLOCKED: pass rate 0.625 is below the threshold 0.700

exit status: 1

Myth vs fact

Myth

One threshold is enough.

Fact

A threshold alone lets a large drop through when the old score was far above it, and an average hides a failed critical case. Use a threshold, a baseline margin and critical cases together.

Myth

If the gate is red, just re-run it.

Fact

Re-running until it passes teaches the team to ignore the gate. Fix flakiness with repeated runs, majority votes and honest thresholds, so a red build means something.

Myth

Evals in CI must call the real model every time.

Fact

Replaying recorded responses (Topic 9.6) tests your prompt assembly and parsing for free on every change, and a live run on a schedule catches model drift.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Store the eval result (rate, per-case results, model version, prompt hash) for every run in a table or file. A trend over weeks shows slow drift that no single run does, and lets you bisect which commit lowered quality.
▸
Estimate the noise before choosing a threshold: run the unchanged prompt 10 times, take the lowest and the spread, and set the threshold below the low end. Otherwise the gate measures luck.
▸
Run evals for pull requests from forks without secrets: a fork must not be able to read your API key. Use the replay mode there and run the live eval after a maintainer approves the merge.
▸
Treat the model name as a pinned dependency. When you change it, expect the whole eval to move; update the baseline deliberately and record why.
▸
In GitHub Actions a scheduled workflow runs from the default branch only, and a pull request from a fork does not receive your secrets. So the nightly live job guards main, and pull requests get the replay job. Recall from Topic 6.8: a live run that hits a rate limit should retry, not fail the gate.

Remember this

1
Why this exists. An eval you must remember to run gets skipped on the busy day. CI (continuous integration) is the server that runs checks automatically on every change (see the CI/CD course at /cicd). Put the eval in CI, make the result a status the pipeline understands (exit code 0 means pass, anything else means fail), and a bad prompt change cannot be merged by accident.
2
Three rules in the gate. First, a threshold: the pass rate must be at least some number (here 0.700). Second, a baseline rule: do not drop more than a small margin below the version in production, because a high threshold alone lets a drop from 0.95 to 0.75 through. Third, critical cases: some cases (the refund request, the safety question) must never fail, whatever the average says. The first example shows prompt v3 passing the average (0.750) and still being blocked because case t1 fails.
3
What triggers a run. Any change that can alter behaviour: files in the prompt folder, the model name or version, the temperature, tool descriptions, retrieval settings, the parsing code. Use path filters in the CI config so documentation edits do not pay for an eval. Also run on a schedule (nightly), because the provider can change a model alias without any change in your repository.
4
Flaky scores. The model's answers vary from run to run, so the same unchanged prompt can score 5 of 6 today and 4 of 6 tomorrow, and a one-run gate blocks good changes at random. Teams then learn to click "re-run" until it passes, and the gate stops meaning anything. Fixes: run each case several times and let the majority decide, set a threshold with a margin below the usual score, lower the temperature, and track the run-to-run spread so you know how big a difference is real. The second example blocks the same unchanged prompt several times with one run, and far less with five.
5
Cost and speed. A full eval with real model calls costs tokens and minutes. Split it: a fast set on every pull request using recorded responses (Topic 9.6) or a small cheap model, and the full live set nightly or before a release. Cap parallel calls to stay under your rate limit (Topics 6.7 and 6.10).
6
Make failures readable. A red build that says FAILED wastes an hour. Print the pass rate, the baseline, which cases failed, which newly broke and what the model said, and save the full report as a CI artifact. The reader should see the problem without rerunning anything. When you add a case after a production incident, fix the prompt in the same pull request.

Explain it without notes

What should an LLM eval gate in CI check, and why is a single threshold not enough?

How do you handle flaky eval scores?

Which changes should trigger an eval run?

How do you keep eval cost and time under control in CI?

Practice

Write gate_status(rate, threshold, failed_critical) returning 1 when the rate is below the threshold or the list failed_critical is not empty, and 0 otherwise. Print the status for (0.80, 0.70, []), (0.65, 0.70, []) and (0.90, 0.70, ["t1"]).

Write majority(passes, needed) where passes is a list of booleans from repeated runs of one case. Return True when at least needed are True. Print it for ([True, True, False, True, False], 3) and ([True, False, False, True, False], 3).

Trade-offs

↔
A strict gate catches more regressions but blocks more good changes; a loose gate does the reverse.
↔
Repeated runs cut flakiness but multiply the cost; repeat only the cases near the pass/fail line.
↔
Replay in pull requests is free and fast but cannot see model drift; live runs see drift but cost money.

Done when you can

Done when you can write a gate with a threshold, a baseline margin and critical cases.
Done when you can end an eval script with an exit status that CI understands.
Done when you can explain flaky scores and reduce them with repeated runs and majority votes.
Done when you can list the changes that should trigger an eval.
Done when you can sketch a pipeline with a fast replay job and a nightly live job (see /cicd).

Command Palette

Words you'll meet

Step by step

01Step 1: turn the eval into a function that returns problems

02Step 2: add the three rules

03Step 3: exit with a status

04Step 4: see the flakiness

05Step 5: wire it into a pipeline

06Step 6: tune the gate by watching it

Try it yourself

Predict the exit status

Move the threshold

More repeats

Code & diagrams

Break it on purpose

Gate on the average only

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can