Production Architecture, High Availability, and Disaster Recovery

1. 1

   Find the single points of failure

   Fill this table for ShopLite's current architecture. Every 'Count = 1' row is a work item.

   | Component        | Count | Failure domain | If it dies...                 | Fix                                   |
   |------------------|-------|----------------|-------------------------------|---------------------------------------|
   | NAT gateway      | 1     | ap-south-1a    | private subnets lose internet | one NAT per AZ                        |
   | RDS PostgreSQL   | 1     | ap-south-1a    | total outage                  | Multi-AZ; tested PITR                 |
   | Redis            | 1     | node           | cache cold, DB overload       | replica + Sentinel / ElastiCache MAZ  |
   | checkout pods    | 2     | same node!     | outage on node drain          | topologySpreadConstraints + PDB       |
   | Argo CD          | 1     | cluster        | deploys stop (apps keep running) | HA install; DR = reinstall + root app |
   | Deploy knowledge | 1     | Ravi           | nobody can release            | runbooks, pipelines, pairing          |

2. 2

   Spread pods across zones

   Replicas on the same node or zone aren't redundancy. Topology spread constraints (Kubernetes course, affinity) force the scheduler to spread them.

       spec:
         topologySpreadConstraints:
           - maxSkew: 1
             topologyKey: topology.kubernetes.io/zone
             whenUnsatisfiable: DoNotSchedule
             labelSelector: { matchLabels: { app: checkout } }
           - maxSkew: 1
             topologyKey: kubernetes.io/hostname
             whenUnsatisfiable: ScheduleAnyway
             labelSelector: { matchLabels: { app: checkout } }

3. 3

   DNS failover between regions (Terraform)

   Route 53 health-checks the primary region's endpoint; when it fails, answers switch to the secondary. Keep the TTL short (60 s) so clients follow quickly. This is the switch for active-passive and warm-standby designs.

   resource "aws_route53_health_check" "primary" {
     fqdn              = "api-mumbai.shoplite.dev"
     type              = "HTTPS"
     resource_path     = "/healthz"
     failure_threshold = 3
     request_interval  = 10
   }
   
   resource "aws_route53_record" "api_primary" {
     zone_id         = var.zone_id
     name            = "api.shoplite.dev"
     type            = "CNAME"
     ttl             = 60
     records         = ["api-mumbai.shoplite.dev"]
     set_identifier  = "primary"
     health_check_id = aws_route53_health_check.primary.id
     failover_routing_policy { type = "PRIMARY" }
   }
   
   resource "aws_route53_record" "api_secondary" {
     zone_id        = var.zone_id
     name           = "api.shoplite.dev"
     type           = "CNAME"
     ttl            = 60
     records        = ["api-singapore.shoplite.dev"]
     set_identifier = "secondary"
     failover_routing_policy { type = "SECONDARY" }
   }

4. 4

   Write the DR runbook, then run it

   A DR plan is a runbook with owners, commands, and a measured time, rehearsed at least twice a year (SRE course, game days). The first rehearsal always finds missing steps.

   # DR: fail over ShopLite from ap-south-1 to ap-southeast-1 (warm standby)
   
   Owner: platform on-call · Target RTO: 30 min · Target RPO: 5 min · Last rehearsal: 2026-08-14 (took 41 min)
   
   1. Declare the incident; incident commander assigned (SRE course, incident command).
   2. Freeze deploys: pause Argo CD auto-sync for prod-mumbai.
   3. Promote the Aurora Global Database secondary in ap-southeast-1 (managed failover) and confirm writes.
   4. Scale prod-singapore: `kubectl --context prod-sg scale deploy --all --replicas=...` / Karpenter limits up.
   5. Point secrets/config at the new DB endpoint (External Secrets refresh).
   6. Route 53: fail over api.shoplite.dev (automatic via health check; force if needed).
   7. Verify: synthetic checkout succeeds; error rate and latency SLOs green.
   8. Communicate status; plan fail-back after the region recovers.

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.