CheckoutErrorBudgetFastBurn — shoplite /checkout error ratio 18% (threshold 7.2%)
Your first page
You're the primary on-call for ShopLite, week one. It's 03:12 and your phone is buzzing.
Briefing
ShopLite runs on ECS behind an ALB with Postgres, exactly as built in the Terraform course, with the SLO burn-rate alerts from the Observability course. You have access to the dashboards, the deploy pipeline, and the team chat. Your secondary on-call is Ravi. Your manager is Meera.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
You're awake, laptop open. The alert is still firing.
- [FIRING] CheckoutErrorBudgetFastBurn severity=critical service=shoplite
- ECS: 4/4 tasks running · ALB: all targets healthy
What's your first move?
Debrief
The strong path: ACKNOWLEDGE (stop escalation), ASSESS impact from user-facing signals, DECLARE and communicate in a shared channel, then MITIGATE by rolling back the change that correlates with the start, and leave root-cause work for daylight. About 30 minutes of impact, most of it before you were paged.
The common failure modes all come from skipping or reordering those steps: acting before knowing impact (restarts), debugging when a rollback was available, and working silently.
The postmortem
Blameless postmortem · draft
Summary. A deploy changed stock reservation to use row locks, causing lock timeouts on concurrent checkouts; rolled back after 32 minutes.
Impact. ~18% of checkout attempts failed between 02:55 and 03:27 (≈1,900 failed checkouts). No data loss or double charges.
Timeline
- 02:50shoplite 1.8.0 deployed
- 02:55Checkout errors begin (lock timeouts)
- 03:12CheckoutErrorBudgetFastBurn pages on-call
- 03:16SEV2 declared; #inc-checkout-errors opened
- 03:20Deploy correlated with error onset; rollback started
- 03:27Error ratio back to baseline; incident mitigated
Root cause. The new reservation code took row locks in a different order from the order-insert path, so concurrent checkouts for the same product waited on each other until the lock timeout.
Contributing factors
- Load tests don't include concurrent checkouts for the same product.
- The deploy went out at 02:50, when few engineers were available and traffic was low enough that canary metrics didn't show the problem.
- No automatic rollback when the error ratio rises after a deploy.
Action items
- preventAdd concurrent same-product checkout scenario to the load test suiteananya
- mitigateAutomatic rollback: pipeline watches checkout error ratio for 15 min after deployplatform
- processDeploy window: no production deploys between 22:00 and 07:00 without an on-call ackmeera
The practice behind it
Mitigate first, debug later
During an incident, the goal is to STOP THE IMPACT, not to understand it. Rollbacks, failovers, feature-flag switches, and scaling are mitigations: they're fast, well-rehearsed, and reversible. Root-cause analysis comes after, when nobody is losing money and people can think. 'We'll roll back, then find out why' is almost always the right call when a recent change correlates with the start.
Declare early, downgrade freely
Declaring an incident costs a minute; not declaring one costs coordination, customer trust, and a timeline. Good teams make declaring easy and blameless ('I think this might be an incident' is enough), and are happy to downgrade or close incidents that turn out small.
Your turn
Write the first message you'd post in the incident channel for this shift. What must it contain?
Interview questions
You're paged at 3 a.m. for elevated errors. Walk me through your first 15 minutes.