An incident detector over a scripted timeline
Three drills of seven minutes each. The detector names the kind of incident and the last deploy before it.
An incident is when the system is failing users right now. LLM apps have four typical ones: provider outage, cost spike, prompt regression and an injection report. A runbook and a practised drill turn panic into a checklist.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write first_alert_minute(error_percents, limit, n) that returns the first minute (0-based) at which the error percent has been at least limit for n minutes in a row, or -1.
Write last_deploy_before(deploys, minute) where deploys is a list of (minute, name); return the name of the latest deploy at or before minute, or "none".
Explain it without notes
Describe how you would tell a provider outage from a prompt regression from a cost spike, using the signals you collect.
What belongs in a runbook for a provider outage?
Why change only one thing at a time during an incident?
Expected output
drill 1: provider outage detected at minute 3; last change: no deploy before it, look outside
drill 2: cost spike detected at minute 3; last change: deploy 41: add 5 examples to the prompt
drill 3: prompt regression detected at minute 3; last change: deploy 42: new answer format