SEV1 — ap-south-1a network degradation; 50% of ShopLite requests failing; RDS primary is in 1a
An availability zone goes dark
Incident commander during a cloud provider zone failure.
Briefing
AWS reports network degradation in ap-south-1a. ShopLite runs tasks across 1a and 1b (Terraform course, Stage 2). The RDS instance is Multi-AZ with its primary in 1a. It's 11:05 on a weekday.
Your shift
Decide before you look. Each choice is locked once made; the next decision unlocks after it. There's no perfect path in real incidents either, only better and worse judgement under uncertainty.
Half the requests fail. The ALB still sends traffic to 1a tasks, which pass health checks intermittently.
- AWS Health: Increased network errors, ap-south-1a
- 5xx by AZ: 1a 94% · 1b 3%
What do you do first?
Debrief
Zone failures are exactly what multi-AZ designs are for, but only if you use them decisively: shift traffic away from the bad zone, fail over stateful services with synchronous standbys, and prefer the smallest scope of failover (zone before region). Practise these steps regularly so they're routine on the day.
The postmortem
Blameless postmortem · draft
Summary. AWS network degradation in ap-south-1a caused request failures until traffic was shifted to 1b and the database failed over.
Impact. ~50% of requests failing for 7 minutes, then intermittent checkout errors for 3 minutes; no data loss.
Timeline
- 11:02AZ degradation begins
- 11:05Incident declared
- 11:11Traffic shifted to 1b
- 11:13RDS forced failover; complete 11:15
- 12:30AWS resolves; rebalance starts
Root cause. Provider-side network degradation in one availability zone.
Contributing factors
- Health checks passed intermittently, so the ALB kept routing to degraded tasks.
- Zone evacuation and DB failover were manual and had never been practised.
Action items
- mitigateEnable ALB zonal shift / automatic target-group AZ removal based on error ratesplatform
- processQuarterly game day: AZ evacuation + RDS failoversre
- detectPer-AZ error-rate alertsre
The practice behind it
RTO, RPO, and failover scope
RTO is how long recovery takes; RPO is how much data you can lose (AWS course, Topic 7.4). Multi-AZ synchronous replication gives near-zero RPO and minutes of RTO for zone failures. Cross-region asynchronous setups trade some RPO for surviving regional failures. Choose the smallest scope that addresses the failure: instance, zone, then region.
Your turn
Why might a health check keep passing on tasks in a degraded zone?
Interview questions
How do you design for and respond to an availability-zone failure?