Topic 11.7
Incident Drills
In one line
An incident is when the system is failing users right now. LLM apps have four typical ones: provider outage, cost spike, prompt regression and an injection report. A runbook and a practised drill turn panic into a checklist.
Think of it like this
A fire drill at school. Nobody can think calmly when the alarm rings for real, so the school practises: everyone knows the exit, the meeting point and who counts heads. The first real fire goes better because the steps are already in people's legs. A runbook is the written drill for an incident.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Incident
- A period when the system is failing or degrading for users and needs action now.
- Runbook
- A short written procedure for handling a particular kind of incident.
- Drill
- A practice run of an incident response, using scripted data or a test environment.
- Rollback
- Returning to the previous version of a prompt, model or code.
- Fallback
- A backup model or behaviour used when the main one fails.
- Degraded mode
- A reduced but working service, such as simpler answers when the main model is down.
- Post-incident review
- A blame-free write-up of what happened and what will change.
- Time to detect
- How long passes between the start of the problem and someone knowing about it.
Step by step
01Step 1: list the incidents you can already name
Four kinds cover most LLM trouble: the provider is down, the cost jumps, the answers got worse, someone reports an injection. Each has a different first signal. The list is short on purpose.
02Step 2: script a timeline
A drill is a list of minutes. Each minute has requests, errors, average input tokens, bad-format count and an optional deploy. Drill 1 has errors jump with no deploy. Drill 2 has input tokens quadruple after a deploy. Drill 3 has bad formats rise after a deploy.
# (requests, errors, avg_tokens_in, bad_format, deploy)
(30, 1, 990, 1, "deploy 41: add 5 examples to the prompt"),
(30, 1, 4200, 0, ""), # input tokens quadrupled03Step 3: write the detector
For each minute, check the rules in order of urgency and return the first minute and the kind. Use whole-number percent maths so every language agrees. A real detector uses windows and several minutes in a row (Topic 11.5); the drill keeps one minute to stay small.
04Step 4: find the suspect change
After detecting, look back from that minute for the most recent deploy. In drills 2 and 3 it names the change. In drill 1 there is none, which is also information: the cause is probably outside your system, so check the provider before touching your code.
05Step 5: follow the runbook, one change at a time
The runbook example prints the steps for a prompt regression and a provider outage and applies one config change per incident (prompt_version back to v8, then model to backup). Printing the config before and after gives you a record of exactly what you changed.
06Step 6: practise and improve
Run the drills with the team. Measure how long it takes to find the right runbook, and fix the places where people hesitated. Each drill should end with one improvement: a clearer alert message, a missing dashboard panel, a safer config switch.
Try it yourself
- 1
Write drill 4
Add a fourth drill where errors reach 40% (below the outage rule) for four minutes. Does the detector see it? What does that tell you about thresholds and about the multi-window rules in Topic 11.5?
- 2
Add an incident to the runbook
Add
injection reportwith three steps and a changesearch_tool=off. Run the incident loop with it first. Why is it good that the change turns something off rather than deleting code? - 3
Time the drill
Run the detector on drill 2 and write down the minute it finds, then the minute the deploy happened. How many minutes of bad service did users get before the detector fired?
Code & diagrams
Three drills of seven minutes each. The detector names the kind of incident and the last deploy before it.
Expected output
drill 1: provider outage detected at minute 3; last change: no deploy before it, look outside
drill 2: cost spike detected at minute 3; last change: deploy 41: add 5 examples to the prompt
drill 3: prompt regression detected at minute 3; last change: deploy 42: new answer formatEach incident has steps and one reversible configuration change. The log shows the config before and after.
Expected output
INCIDENT prompt regression
config before: model=primary, prompt_version=v9, max_input_tokens=8000
step 1: compare the last two prompt versions
step 2: roll back to the previous version
step 3: add the failing cases to the eval set
config after: model=primary, prompt_version=v8, max_input_tokens=8000
INCIDENT provider outage
config before: model=primary, prompt_version=v8, max_input_tokens=8000
step 1: check the provider status page
step 2: switch traffic to the backup model
step 3: tell support
config after: model=backup, prompt_version=v8, max_input_tokens=8000Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
An outage threshold that is too high
In detect (Python), change errors * 100 // requests >= 50 to errors * 100 // requests >= 99.
Myth vs fact
Myth
Incidents mean someone made a mistake.
Fact
Most incidents come from many small causes in a complex system. Blame-free reviews find fixes; blame hides the facts.
Myth
A bigger model or a better prompt prevents incidents.
Fact
Providers have outages and prompts regress. Preparation (fallbacks, rollback, alerts, runbooks) is what shortens them.
Myth
Runbooks are paperwork.
Fact
A runbook is the memory of the person who was awake last time. It saves minutes when minutes cost users.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Make rollbacks one action: prompts, model names and limits live in configuration with versions (Topic 12.7), so "go back to v8" is a config change and not a deploy.
- ▸
Have a degraded mode designed in advance: cached answers, a simpler model, a plain "try again soon" message, or a human queue. Users forgive a clear degraded mode more than a hang.
- ▸
Track time to detect, time to mitigate and time to resolve for each incident. Detection time is where observability pays: it falls when alerts and dashboards are right.
Remember this
- 1
Why this exists. At 3 a.m. with an alert firing, people make mistakes: they change three things at once, forget who to tell, or fix the symptom and lose the evidence. A runbook gives a short list of steps for each kind of incident, and a drill is a practice run, so the first time you follow it is not the first time it matters. Hectal's SRE course (/sre) covers incident response, on-call and post-incident reviews in depth; this topic is the LLM-specific part.
- 2
Provider outage. The model API returns errors or times out for everyone. Signals: error rate jumps within a minute, mostly 5xx and timeouts, with no deploy before it. Actions: check the provider's status page and switch traffic to a fallback model or provider (Topic 6.9). Cap retries so you do not make it worse (Topic 6.8). Tell support and users, and show the degraded-mode message. Evidence to keep: the time it began and the error types.
- 3
Cost spike. Spending per request or per hour jumps. Signals: tokens in per request rises (here from 1,000 to 4,200 after a deploy) while errors stay normal. Likely causes: a longer prompt (more examples, more retrieved text), a loop that retries or calls the model repeatedly, or one user or feature suddenly using much more. Actions: find the last change, cap input tokens, roll back if the cap is not enough, and check the per-user usage.
- 4
Prompt regression. Answers got worse after a change to the prompt, the model or retrieval. It does not crash: you see it as more format failures or more thumbs-down. Signals: the bad-format rate or the downvote rate rises right after a deploy. Actions: compare the last two prompt versions, roll back to the previous one (this is why prompts are versioned configuration: Topic 12.7), and add the failing cases to the golden set so the next change is tested against them.
- 5
Injection report. A user or researcher reports that the bot followed instructions hidden in a document or message (Phase 10). Actions: record the exact request id and input, switch the affected tool to read-only or off, add the case to your red-team suite (Topic 10.8), fix the guard, and check logs for similar requests. Treat the report as a security issue: limit who sees the details until it is fixed.
- 6
Detect, then change one thing. The first code example is a detector. For each minute it checks three simple rules (error rate at least 50%, input tokens at least double the baseline, bad format at least 20%). It names the kind of incident and shows the last deploy before it. Drill 1 has no deploy before the failure, so the answer is "look outside" (the provider). Drills 2 and 3 point to a deploy. The second example runs a runbook as data. Each incident has steps and one reversible configuration change, and the log prints the config before and after.
- 7
After the incident. Write a short review without blame. Say what happened, how you noticed, and how long it took to detect and to fix. Add what made it worse, and one or two changes (an alert, a limit, a test). The aim is that the same incident is detected sooner next time.
Explain it without notes
Describe how you would tell a provider outage from a prompt regression from a cost spike, using the signals you collect.
What belongs in a runbook for a provider outage?
Why change only one thing at a time during an incident?
Practice
Write first_alert_minute(error_percents, limit, n) that returns the first minute (0-based) at which the error percent has been at least limit for n minutes in a row, or -1.
Write last_deploy_before(deploys, minute) where deploys is a list of (minute, name); return the name of the latest deploy at or before minute, or "none".
Trade-offs
- ↔
Automatic rollback is fast and can roll back the wrong thing; many teams automate the switch and keep a human on the decision.
- ↔
Tight detection thresholds find incidents early and raise false alarms; tune with replayed history.
- ↔
Drills cost team time and make real incidents cheaper.
Done when you can
Done when you can name the four LLM incident types and the first signal of each.
Done when you can write a detector for an outage, a cost spike and a regression.
Done when you can write a runbook with ordered steps and one reversible change.
Done when you can explain what a blame-free review contains.