SEV2 ongoing — product pages 40% slower since a config change; VP Engineering has joined the channel
The VP wants a root cause. Now.
You're incident commander for an ongoing incident; a senior leader has just joined and is asking questions.
Briefing
At 14:05 a feature flag enabling the new recommendation widget was turned on for 100% of users. Product-page p99 went from 180 ms to 2.4 s. It's 14:22. You suspect the flag but aren't certain: a database maintenance job also started at 14:00.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
You have two suspects: the flag rollout at 14:05, and a vacuum job on Postgres since 14:00.
- /products p99: 0.18s → 2.4s (step change at 14:06)
- postgres CPU 38% (normal: 30%) · no lock waits
- flag recommendations_v2: 100% since 14:05
What do you do?
Debrief
Prefer the mitigation that's also a diagnostic: reversible, quick, and targeted at the change that correlates with the onset. Handle stakeholders with facts, a plan, and a date, and deflect blame towards systems. Close incidents deliberately, with a resolution update and a scheduled postmortem.
The postmortem
Blameless postmortem · draft
Summary. Enabling recommendations_v2 for 100% of users introduced an N+1 query per product page, raising p99 from 180 ms to 2.4 s; mitigated by disabling the flag.
Impact. All product page views were slow (p99 2.4 s) for 19 minutes; conversion dipped ~9% in that window.
Timeline
- 14:05recommendations_v2 enabled at 100%
- 14:06/products p99 rises to 2.4 s
- 14:14Latency SLO alert fires; incident declared at 14:16
- 14:24Flag disabled; latency recovers by 14:26
- 14:40Incident resolved
Root cause. The widget issued one database query per recommended product (N+1), multiplying query volume per page view.
Contributing factors
- Feature flags could be enabled directly at 100% with no gradual rollout.
- The widget's load test used a catalogue with 5 products instead of production's 200.
Action items
- preventFlag system: enforce staged rollout (1% → 10% → 50% → 100%) with automatic latency checksplatform
- preventFix N+1 in recommendations widget; add query-count assertion to its testsdiscovery-team
- detectAlert on DB queries per request (span metrics) exceeding 3× baselinesre
The practice behind it
The role of the incident commander
The IC coordinates; they don't have to fix things personally. Their job is keeping the response focused on mitigation, deciding when to escalate, making sure communication goes out on schedule, and shielding responders from distraction, including senior leaders who want answers. On small incidents one person is both IC and responder; the moment it gets busier, split the roles.
Your turn
Rewrite the VP's request into the question a blameless postmortem should answer.
Interview questions
How do you handle pressure from leadership during an incident?