Topic 11.9
Incident Drills
In one line
Four things go wrong in every RAG service sooner or later: stale answers, an empty index after a bad deploy, a leak report and a cost spike. For each, write a runbook now: how to notice, what to check first and what to do, in priority order.
Think of it like this
A school holds fire drills. Nobody expects a fire on a Tuesday, but when the alarm rings, nobody wants to discuss what to do. The children walk out in a known order. A runbook is a fire drill for your service: the steps written down, tried in advance, so that at 3 in the morning you follow a list instead of inventing a plan.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Runbook
- A written list of steps to follow for a known failure.
- Incident
- An event that hurts users or the business and needs a response.
- On-call
- The person responsible for responding to alerts out of hours.
- Baseline
- The normal level of a metric in a quiet period, used to judge a change.
- Canary question
- A fixed question with a known right answer, asked regularly to check the service works.
- Post-incident review
- A write-up of what happened and how to prevent it, focused on systems, not blame.
- Priority
- The order of action when several things are wrong. Harm to people first.
Step by step
01Step 1: list the failures you expect
Start with the four above, then add what your own history shows. For each: how would we notice, and how quickly?
02Step 2: pick a signal for each
A number you already collect (Topic 11.6): index lag, indexed document count, refusal rate, cross-tenant hits, cost per answer. Compare it with its baseline, not with a guess.
03Step 3: write the first check and the first safe action
The first check is the cheapest thing that splits the likely causes. The first action is the one that stops the harm and can be undone. Keep the full fix for after the harm has stopped.
if m["index_docs"] * 2 < m["docs_expected"]:
alerts.append(("shrunken index", "the last deploy and the index version", "switch back to the previous index version"))04Step 4: order by harm
Leaks first, then wrong answers (empty index, stale), then money. The example's "empty index" snapshot raises two alerts; the cause comes before the symptom.
05Step 5: drill it
In a test environment, empty the index on purpose, or add a document that leaks, and time how long a person takes to follow the steps. Fix the steps that confuse them.
06Step 6: add a canary question
Ask a fixed question every minute and check that the right document is found (the Build topic's health() does this). A canary catches an empty or broken index before any user does.
Try it yourself
- 1
Add a snapshot
Add a snapshot with a stale index and a cost spike at once. Which alert is listed first, and why is that the right order?
- 2
Tune a threshold
Change the cost rule from double the baseline to 1.5 times. Which snapshots change? What is the cost of a rule that fires too easily?
- 3
Write a new rule
Add a rule for a canary question that fails (a new key
canary_okwith 0 or 1). Where in the priority order does it belong?
Code & diagrams
The thresholds (half the expected documents, 48 ticks, double the cost, 30 percent refusals) are example values: set yours from your own baselines.
Expected output
healthy: no alerts
stale: 1 alert(s)
1. stale answers | check first: queue backlog and worker errors | then: restart or add workers, show the as-of date
empty index: 2 alert(s)
1. shrunken index | check first: the last deploy and the index version | then: switch back to the previous index version, then fix the job
2. refusals up | check first: whether the index is small or the threshold changed | then: follow the shrunken index steps if the index is small
leak report: 1 alert(s)
1. leak | check first: which filter or cache key is missing the tenant id | then: switch off the answer cache and shared search, keep the logs, tell the security owner
cost spike: 1 alert(s)
1. cost spike | check first: prompt size, cache hit rate and retries | then: lower k and the output limit, turn the cache onBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Lose the shrunken-index rule
In the Python tab, replace if m["index_docs"] * 2 < m["docs_expected"]: with if False: and run.
Myth vs fact
Myth
A runbook is paperwork for big companies.
Fact
A page of steps for each of four failures costs an afternoon and saves hours at the worst moment. Small teams need it more, because the same person is on call for everything.
Myth
If cost drops, things are fine.
Fact
A broken index refuses cheaply, so cost falls while users get nothing. Watch quality and cost together.
Myth
Investigate first, then act.
Fact
For a leak, stop the exposure first (switch off the cache, shared search), then investigate. Evidence is kept in the logs.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep the previous index version reachable by a name (an alias) so rollback is a single change. Vector stores and search engines mostly support aliases or collection swaps (Topic 5.4 covers collections).
- ▸
Use a kill switch per feature (cache, re-ranker, shared index) that can be turned off without a deploy. The best incident response is a switch that was built in advance.
- ▸
After a leak, check every place text lives: the index, the cache, the logs, the traces, backups and the evaluation sets (Topic 10.6). Fixing only the index leaves copies behind.
- ▸
Track time to detect and time to resolve for each incident class. If detection takes days, the alert is the first thing to improve.
- ▸
The Observability course (
/observability) and the SRE course (/sre) cover alert design, on-call practice and post-incident reviews in depth.
Remember this
- 1
Why drills. Under pressure people skip steps, check the wrong thing first, or fix the symptom and leave the cause. A runbook gives the first check and the first safe action for each known failure, and a drill finds out which steps are missing before a real incident does. LLM Engineering Topic 11.7 drilled model-call incidents (provider outage, prompt regression); these four are specific to retrieval.
- 2
Incident 1: stale answers. Signal: index lag above the allowance (Topic 11.2), or a user reports an old price. First check: the ingestion queue's backlog and the workers' error log. Action: restart or add workers; if the source is changing faster than you index, widen the "as of" notice. Afterwards: add an alert on the age of the oldest unindexed change, and a re-check job that repairs missed events.
- 3
Incident 2: an empty or shrunken index after a bad deploy. Signal: refusal rate jumps and the number of indexed documents drops (the example's 96 percent refusal and 0 documents). First check: the last deploy and the index version, and whether the job read an empty or wrong source. Action: switch back to the previous index version (keep the old one for exactly this reason, Topic 11.8), then fix the job. Note that the cost per answer falls in this incident, because refusals are cheap: a cost dashboard alone would call it good news.
- 4
Incident 3: a leak report. Signal: a user sees content they should not, or a check finds hits from another tenant. This is the highest priority of the four. First check: which filter or cache key is missing the tenant or permission (Topics 10.3 and 10.4); caches and logs leak as easily as the index. Action: switch off the answer cache and shared search paths, keep the evidence, tell the security owner, and only then investigate. Defensive rule: remove the exposure first, understand it second.
- 5
Incident 4: a cost spike. Signal: cost per answer more than double its baseline (the example uses 5,200 micro-dollars against 1,700; a micro-dollar is one millionth of a dollar, so 1,700 is 0.0017 dollars). First check: prompt size per answer, cache hit rate and the retry rate. Action: lower k and the output limit, turn the cache on, and find the change that caused it: a new document type with very long passages, a loop of retries, a prompt change. Topic 11.5's sum tells you which part grew.
- 6
Write the runbook as data, and drill it. The example encodes each rule with its priority, first check and first action, and prints them for five metric snapshots. Order matters: if two alerts fire together (an empty index also raises the refusal rate), the list shows the cause first. Run a drill every quarter: pick a failure, break a test environment on purpose, and see whether the on-call person can follow the steps. Afterwards write a short post-incident review: what happened, how it was noticed, what fixed it, what stops it happening again. No blame.
Explain it without notes
Why write runbooks and run drills for a RAG service?
How do you handle an index that became empty after a deploy?
A user reports they saw another customer's content in an answer. What do you do, in order?
Why can cost per answer fall during an incident?
Practice
Write first_step(alert, runbook) returning the first step from a dict of alert name to list of steps, or the text no runbook: write one when the alert is unknown. Print it for a known and an unknown alert.
Write canary_ok(search, question, expected_id) that calls search(question) (it returns a list of ids) and returns True when the first id is the expected one. Test it with two small fake search functions.
Trade-offs
- ↔
Sensitive alerts: early warning, and more false alarms and tired on-call people.
- ↔
Automatic rollback: fast recovery, and a risk of flapping if the signal is noisy.
- ↔
Kill switches: quick safety, and extra code paths that must be tested.
Done when you can
Done when you can name the four retrieval incidents and the signal for each.
Done when you can write a first check and a first safe action for each.
Done when you can order simultaneous alerts by harm.
Done when you can explain why a leak is handled by stopping exposure first.
Done when you can describe a drill and a post-incident review.