A runbook as data
Each incident has steps and one reversible configuration change. The log shows the config before and after.
An incident is when the system is failing users right now. LLM apps have four typical ones: provider outage, cost spike, prompt regression and an injection report. A runbook and a practised drill turn panic into a checklist.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write first_alert_minute(error_percents, limit, n) that returns the first minute (0-based) at which the error percent has been at least limit for n minutes in a row, or -1.
Write last_deploy_before(deploys, minute) where deploys is a list of (minute, name); return the name of the latest deploy at or before minute, or "none".
Explain it without notes
Describe how you would tell a provider outage from a prompt regression from a cost spike, using the signals you collect.
What belongs in a runbook for a provider outage?
Why change only one thing at a time during an incident?
Expected output
INCIDENT prompt regression
config before: model=primary, prompt_version=v9, max_input_tokens=8000
step 1: compare the last two prompt versions
step 2: roll back to the previous version
step 3: add the failing cases to the eval set
config after: model=primary, prompt_version=v8, max_input_tokens=8000
INCIDENT provider outage
config before: model=primary, prompt_version=v8, max_input_tokens=8000
step 1: check the provider status page
step 2: switch traffic to the backup model
step 3: tell support
config after: model=backup, prompt_version=v8, max_input_tokens=8000