SEV1 — support reports customers seeing duplicate charges on their cards for single orders; social media posts appearing
Customers were charged twice
Incident commander for an incident with direct financial impact on customers.
Briefing
Since this morning's deploy (09:40), some customers have been charged twice for one order. Nothing is 'down': dashboards are green, error rates normal. It's 11:15, and support has 40 tickets.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
Everything looks healthy. The only signal is customer reports.
- support: 40 tickets 'charged twice', all orders after ~09:45
- checkout error ratio 0.2% (normal) · payments /charge calls per checkout: 1.31 (normal: 1.00)
What do you do first?
Debrief
Correctness incidents are as serious as outages, often more so, and they're easy to miss because dashboards stay green. Stop the harm first (roll back), communicate honestly and specifically, then remediate precisely with verified data, peer review, and controlled batches. Afterwards, add monitoring for correctness signals like charges per order.
The postmortem
Blameless postmortem · draft
Summary. A deploy added payment retries without idempotency keys, causing duplicate charges for ~31% of checkouts for 1 h 36 min.
Impact. 612 customers double-charged (refunded within 2.5 hours); 83 support tickets; public social media posts.
Timeline
- 09:40Deploy with new payment retry logic
- 09:45First duplicate charges
- 11:15Support escalation; SEV1 declared
- 11:21Rollback complete; duplicates stop
- 11:35Public status update posted
- 14:00All 612 duplicate charges refunded
Root cause. Retries of timed-out payment calls lacked an idempotency key, so the provider processed both attempts as separate charges.
Contributing factors
- No monitoring of charges per order (a correctness SLI).
- Retry logic was reviewed for correctness of the code, not of the payment semantics.
- Detection relied on customer reports.
Action items
- preventIdempotency keys on all payment calls; provider-side dedup verified in testsshoplite
- detectAlert when payments per order exceeds 1.02 over 10 minutessre
- mitigateRemediation runbook: reconciliation query, 2-person review, batched refundssre + finance
The practice behind it
Correctness is a reliability property
Availability and latency SLOs don't cover a system that is up, fast, and wrong. For money, inventory, and personal data, define correctness SLIs (charges per order, reconciliation mismatches, duplicate messages) and alert on them. These incidents are rarer but often do more damage, because they break trust rather than just being inconvenient.
Your turn
Write the public status update for this incident at 11:35.
Interview questions
How do you handle an incident where the system is 'up' but producing wrong results for customers?