SEV1 — shoplite, search, and account pages all timing out; only payments was 'slightly slow' ten minutes ago
One slow dependency, everything down
Incident commander for a spreading outage.
Briefing
The payment provider degraded at 16:02 (responses taking 20 s). By 16:10, pages that never touch payments are failing too. ShopLite's services share one database connection pool and one HTTP client pool per task.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
Everything is failing, but only payments was slow originally.
- shoplite in-flight requests: 1,240 (normal 40) · DB pool: 10/10 in use, 380 waiting
- timeout acquiring connection from pool after 30000ms
- payment provider status: degraded performance
What's happening, and what do you do first?
Debrief
Cascading failures spread through shared resources such as connection pools, threads, and queues, held by requests waiting on something slow. Stop the spread first by failing fast at the slow dependency, then restore degraded functionality where the business allows, then isolate dependencies so the next slow one can't take everything down.
The postmortem
Blameless postmortem · draft
Summary. Payment provider latency exhausted shared database connections, cascading into a full ShopLite outage.
Impact. All pages degraded or failing for 10 minutes; checkout degraded for 46 minutes.
Timeline
- 16:02Provider latency rises to ~20 s
- 16:10Connection pool exhausted; site-wide failures
- 16:14Payment circuit breaker forced open; site recovers
- 16:22Checkout moves to 'payment pending' queue mode
- 16:48Provider recovers; queue drained
Root cause. Checkout requests held database connections while waiting on a slow external payment call with a long timeout, exhausting the shared pool.
Contributing factors
- Single shared pool for all request types.
- 5-second payment timeout (raised to 30 s last year after a false-positive incident).
- Circuit breaker existed but was manual only.
Action items
- preventRelease DB connections before external calls; per-dependency pools (bulkheads)shoplite
- mitigateAutomatic circuit breakers with 2 s timeouts on external callsshoplite
- processMonthly test of the pending-payment queue pathsre
The practice behind it
Timeouts, bulkheads, circuit breakers
TIMEOUTS bound how long any call can hold resources. BULKHEADS partition resources (pools, threads) so one dependency's failure can only exhaust its own share, like watertight compartments in a ship. CIRCUIT BREAKERS stop calling a failing dependency for a while, giving it room to recover and giving callers instant, predictable failures. Together they turn 'everything fails' into 'one feature degrades'.
Your turn
How can you spot a cascading failure from metrics, before reading any logs?
Interview questions
What is a cascading failure and how do you design against it?