SEV1 — shoplite, payments, and search all returning 5xx; status page shows full outage
Fourteen people in the channel
You're the on-call SRE and the first responder in a channel that fills up fast.
Briefing
At 19:02 every ShopLite service starts returning errors. Within four minutes, fourteen people have joined #inc-outage, including three team leads, two directors, and the CTO. Messages are flying: theories, screenshots, 'is anyone looking at X?'.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
The channel is chaos. Two engineers are independently restarting services. Nobody has said who's in charge.
- dev-lead-1: is it the DB?? director: what's the ETA dev-2: restarting payments dev-5: I'm restarting shoplite cto: who's running this?
What do you do?
Debrief
In a large incident, the most valuable thing you can do is create structure: one commander, named roles, a single channel, a freeze on uncoordinated changes, and regular updates. With structure in place, the fix (reverting a config change) was found and applied in about 15 minutes.
The postmortem
Blameless postmortem · draft
Summary. A VPC DNS resolver configuration change broke internal hostname resolution, causing a full outage of all ShopLite services for 21 minutes.
Impact. All customer-facing services unavailable from 19:00 to 19:21; ~6,300 failed checkouts; status page updated at 19:08.
Timeline
- 19:00Resolver configuration change applied by network team
- 19:02Errors across all services; alerts fire
- 19:06IC established; roles assigned; change freeze
- 19:14DNS resolution failures identified
- 19:21Change reverted; recovery
- 19:40Resolved; handover to next on-call
Root cause. The new resolver rule forwarded the internal zone to an endpoint that didn't serve it, so lookups for internal hostnames failed.
Contributing factors
- Network changes were applied to all VPCs at once, with no canary environment.
- No synthetic DNS check or alert on resolution failures.
- Two uncoordinated restarts in the first minutes caused additional connection churn.
Action items
- preventApply resolver changes to staging, then one production VPC, with automated resolution checksnetwork
- detectBlackbox DNS probes for critical internal names (Observability course, Case 5.3)sre
- processIncident command training for all on-call engineers; IC checklist pinned in the incident templatemeera
The practice behind it
Incident command roles
Adapted from emergency services' Incident Command System: the INCIDENT COMMANDER owns the response and makes decisions; the OPERATIONS LEAD directs the technical work; the COMMUNICATIONS LEAD handles status pages, support, and leadership; a SCRIBE keeps the timeline. Roles are about function, not seniority, and can be handed over explicitly ('I'm handing IC to Ravi, Ravi please confirm').
Your turn
Draft the stakeholder update the comms lead should post at 19:10, before the cause is known.
Interview questions
How would you run a major incident with many people involved?