Game day plan review — 'kill the payments service in production and see what happens'
Breaking things on purpose
You're reviewing a colleague's first chaos-engineering plan.
Briefing
After Shift 2.2's cascade, the team wants to verify the new circuit breakers. Arjun proposes stopping the payments service in production at noon on Friday and watching what happens.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
Arjun's plan is one line: 'Kill payments in prod, observe.'
What's the most important change to the plan?
Debrief
Chaos engineering is the scientific method applied to reliability: a hypothesis about steady-state behaviour, a controlled injection, a limited blast radius, abort criteria, and honest recording of surprises. Start small and in safer environments, and treat the unexpected finding as the win.
The postmortem
Blameless postmortem · draft
Summary. Game day: payments failure injected; checkout degraded correctly; a hidden dependency in account pages was discovered.
Impact. Account page p99 elevated to 900 ms for 3 minutes during the experiment.
Timeline
- WedStaging run found a broken fallback page; fixed
- Fri 12:00Production injection (one task)
- Fri 12:03Unexpected account-page latency; experiment aborted
Root cause. Account page 'saved cards' widget called payments synchronously with no timeout.
Contributing factors
- Dependency not listed in the service's documented dependencies.
Action items
- preventAdd timeout and fallback to the saved-cards callaccounts-team
- processQuarterly game days with a rotating failure scenariosre
- detectGenerate dependency maps from traces (service graphs) instead of documentationsre
The practice behind it
Principles of chaos engineering
Define steady state in terms of measurable user-facing behaviour; hypothesise that it holds during a failure; introduce realistic failures (instance loss, latency, dependency errors); minimise the blast radius; and automate experiments over time. The goal is building confidence in the system's behaviour, not causing outages.
Your turn
Write a hypothesis and abort criteria for an experiment that adds 500 ms latency to the database for 10% of ShopLite tasks.
Interview questions
How would you introduce chaos engineering to a team?