A runbook as data: five snapshots, their alerts in order
The thresholds (half the expected documents, 48 ticks, double the cost, 30 percent refusals) are example values: set yours from your own baselines.
Four things go wrong in every RAG service sooner or later: stale answers, an empty index after a bad deploy, a leak report and a cost spike. For each, write a runbook now: how to notice, what to check first and what to do, in priority order.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write first_step(alert, runbook) returning the first step from a dict of alert name to list of steps, or the text no runbook: write one when the alert is unknown. Print it for a known and an unknown alert.
Write canary_ok(search, question, expected_id) that calls search(question) (it returns a list of ids) and returns True when the first id is the expected one. Test it with two small fake search functions.
Explain it without notes
Why write runbooks and run drills for a RAG service?
How do you handle an index that became empty after a deploy?
A user reports they saw another customer's content in an answer. What do you do, in order?
Why can cost per answer fall during an incident?
Expected output
healthy: no alerts
stale: 1 alert(s)
1. stale answers | check first: queue backlog and worker errors | then: restart or add workers, show the as-of date
empty index: 2 alert(s)
1. shrunken index | check first: the last deploy and the index version | then: switch back to the previous index version, then fix the job
2. refusals up | check first: whether the index is small or the threshold changed | then: follow the shrunken index steps if the index is small
leak report: 1 alert(s)
1. leak | check first: which filter or cache key is missing the tenant id | then: switch off the answer cache and shared search, keep the logs, tell the security owner
cost spike: 1 alert(s)
1. cost spike | check first: prompt size, cache hit rate and retries | then: lower k and the output limit, turn the cache on