Topic 13.7
Playbook: The RAG Launch Checklist
In one line
The checklist to run before real users arrive, written as a gate that reports blockers, and the alerts to watch every day of the first week.
Think of it like this
The pre-flight checklist. Pilots who have flown a thousand hours still read it, item by item, aloud. A checklist is not for what you do not know. It is for what you will forget when you are busy and the launch is today.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Launch gate
- A set of measured checks that must all pass before release.
- Blocker
- A failing check that stops the launch.
- Shadow mode
- Running the new system on real traffic without showing its answers to users.
- Kill switch
- A control that turns a feature off quickly and safely.
- Baseline cost per question
- Day one's cost per question, the reference for spotting a jump.
- Spend limit
- A cap on money or tokens per day that stops a runaway cost.
Step by step
01Step 1: list the checks as numbers
For each area write what you measure, the direction (at least or at most) and the limit. A check you cannot measure cannot gate a launch.
02Step 2: run the gate
The loop prints PASS or FAIL per check, collects the blockers and prints YES or NO for the launch. Run it the day before, and again on launch day.
03Step 3: launch in steps
Shadow, a small share, everyone. At each step look at the week-one numbers before widening.
04Step 4: compute the daily numbers
Per day: refusal rate, thumbs-down rate, p95 and cost per question compared with day one. Keep the limits in constants at the top.
05Step 5: act on the alerts
A refusal spike points at ingestion or the threshold. A thumbs-down spike points at answer quality. A latency jump points at a stage budget. A cost jump points at prompt size or retries.
06Step 6: feed the golden set
Every alert day yields questions. Add them with the expected passage so the next release is tested against what real users asked.
Try it yourself
- 1
Fix the blockers
In the gate, change the two failing values to passing ones and run. The launch line should say YES. Which real work would each change stand for?
- 2
Add a check
Add a security check 'restricted passages returned to outsiders in the test suite' with value 0 and limit 0, and a failing version at value 2. See how the blockers list changes.
- 3
Tighten an alert
In the week-one example, set MAX_REFUSAL_RATE to 0.10 and run. Which extra days alert? Is that too noisy to be useful?
Code & diagrams
Two checks fail on purpose: three documents have no access label, and ingestion lags 45 minutes against a 30-minute limit. The limits are examples.
Expected output
[PASS] quality: hit rate at k=3 on the golden set 0.91 (needs >= 0.85)
[PASS] quality: unanswerable questions refused 1.00 (needs >= 0.95)
[PASS] quality: answers with a valid citation 0.98 (needs >= 0.95)
[PASS] security: cross-tenant leaks in the test suite 0.00 (needs <= 0.00)
[FAIL] security: documents without an access label 3.00 (needs <= 0.00)
[PASS] cost: dollars per 1000 questions 1.24 (needs <= 1.50)
[PASS] cost: daily spend limit set (1 = yes) 1.00 (needs >= 1.00)
[PASS] operations: p95 latency in seconds 5.20 (needs <= 6.00)
[FAIL] operations: ingestion lag in minutes 45.00 (needs <= 30.00)
[PASS] operations: rollback rehearsed (1 = yes) 1.00 (needs >= 1.00)
blockers: 2
- security: documents without an access label
- operations: ingestion lag in minutes
launch: NOCounts and dollars are example numbers. In production they come from the traces and metrics of Topic 11.6.
Expected output
day 1: ok
day 2: ok
day 3: ok
day 4: refusal rate 0.20 (limit 0.15)
day 5: thumbs-down rate 0.12 (limit 0.10)
day 6: p95 6.8 s (limit 6.0)
day 7: cost per question x2.14 of day 1 (limit x2.00)
alerts on days: 4, 5, 6, 7Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Compare the wrong way round
In the Python gate, change ok = value >= limit if op == ">=" else value <= limit to ok = value >= limit, so every check is treated as 'at least'.
Myth vs fact
Myth
If tests pass, launch day is routine.
Fact
Real users ask what you did not expect. The first week's questions are better test data than any you can invent.
Myth
A checklist is bureaucracy.
Fact
It is the cheapest tool against forgetting the dull, important item, such as a spend limit or a rollback rehearsal.
Myth
Alerts should fire on any change.
Fact
Too many alerts get ignored. Set limits from observed variation and review them after week one.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Gate on the lowest group, not the average: hit rate per question type and per tenant. An average can hide a damaged group (Topic 13.4).
- ▸
Rehearse the rollback. A rollback that has never been run is a hope. Time it and write the steps down (Topic 11.8).
- ▸
After week one, shorten the loop: pull the sampled thumbs-down into the golden set weekly, rerun the gate in CI on every change and keep the limits in version control (Topic 9.7).
Remember this
- 1
Recipe: the launch gate. When to use it: the day before real users. It guards against shipping on a feeling. Write each check as a measured value, a direction and a limit, and let code list the blockers. Four areas: quality (hit rate on the golden set, refusal of unanswerable questions, valid citations), security (zero cross-tenant leaks in tests, every document labelled), cost (dollars per 1000 questions and a daily spend limit) and operations (p95 latency, ingestion lag, a rehearsed rollback). Adapt the limits to your product; never skip an area.
- 2
Recipe: yes/no facts as numbers. When to use it: for checks that are not metrics, such as 'spend limit set'. Store them as 1 or 0 with limit 1 so the same loop handles everything. It guards against a checklist that is half a document and half a script.
- 3
Recipe: the week-one dashboard. When to use it: from the first day of real traffic. Compute per day the refusal rate, the thumbs-down rate, p95 latency and cost per question relative to day one. Alert when one crosses its limit. It guards against the slow failures that no test catches: real users ask different questions from your golden set.
- 4
Recipe: read thumbs-down by reason. When to use it: every day in week one. Sample thumbs-down answers, name the failing stage (Topic 13.4) and add each to the golden set (Topic 9.8). It turns the first week into the best source of hard test cases you will ever get.
- 5
Recipe: roll out in steps. When to use it: always. Shadow mode (run, do not show), then a small share of users, then everyone, with a kill switch that routes to a person or to plain search. It guards against a regression reaching everyone at once (Topic 11.9).
- 6
Recipe: the incident basics. When to use it: before you need it. Have a runbook for stale answers, an empty index, a leak report and a cost spike, and know who is on call (Topic 11.9). It guards against inventing a process during an outage.
Explain it without notes
What are the four areas of the launch gate and one check from each?
Why express yes/no facts as numbers in the gate?
Which signals do you watch in the first week, and what does each point to?
Why launch in steps instead of all at once?
Practice
Write day_alerts(questions, refused, limit) returning 'alert' when refused / questions is greater than limit and 'ok' otherwise. Print the result for two days.
Write cost_ratio(day, baseline_day) that returns the cost per question of day divided by the baseline's, where each is a (questions, dollars) pair. Print it with two decimals for (1400, 3.6) against (1000, 1.2).
Trade-offs
- ↔
A strict gate: fewer bad launches and more delays. A loose gate: faster launches and more incidents.
- ↔
Many alerts: nothing slips by and people stop reading them. Few alerts: calm and some problems found late.
- ↔
A slow staged rollout: safer and slower to deliver value. A fast one: quicker and with a larger blast radius.
Done when you can
Before you ship: the gate runs from code and every area has at least two measured checks.
Before you ship: a spend limit exists and the kill switch and rollback have been rehearsed.
Before you ship: a cross-tenant leak test passes and every document has an access label.
Before you ship: week-one alerts for refusal, thumbs down, p95 and cost per question are set, with an owner.
Before you ship: someone will read the thumbs-down answers daily and add them to the golden set.