Chaos Engineering

1. 1

   Install Chaos Mesh in a test cluster

   Install into its own namespace. Restrict which namespaces it may target with controllerManager.enableFilterNamespace and an annotation on target namespaces.

   terminalcopy cmd

   $ helm repo add chaos-mesh https://charts.chaos-mesh.org

   helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh --create-namespace --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock --set controllerManager.enableFilterNamespace=true

   kubectl annotate ns shoplite-staging chaos-mesh.org/inject=enabled

   ── expected output ──

   NAME: chaos-mesh

   STATUS: deployed

2. 2

   Experiment 1: kill a checkout pod

   Hypothesis: with 3 replicas, readiness probes, and a PDB, killing one pod doesn't affect checkout success. Watch your SLO dashboard (Observability course) while it runs.

   apiVersion: chaos-mesh.org/v1alpha1
   kind: PodChaos
   metadata:
     name: kill-one-checkout
     namespace: shoplite-staging
   spec:
     action: pod-kill
     mode: one                     # blast radius: one pod
     selector:
       namespaces: [shoplite-staging]
       labelSelectors: { app: checkout }

   terminalcopy cmd

   $ kubectl apply -f chaos/pod-kill-checkout.yaml

   kubectl -n shoplite-staging get pods -l app=checkout -w

   ── expected output ──

   checkout-6d8f9c7b5-x4k2p 1/1 Terminating 0 3h

   checkout-6d8f9c7b5-p7r2t 0/1 Pending 0 0s

   checkout-6d8f9c7b5-p7r2t 1/1 Running 0 24s

3. 3

   Experiment 2: slow down a dependency

   Add 800 ms latency to traffic from checkout to the payments service for 5 minutes. Hypothesis: checkout's 2 s timeout and circuit breaker keep p95 under 1.5 s and no threads pile up. This one often fails the first time, which is the point.

   apiVersion: chaos-mesh.org/v1alpha1
   kind: NetworkChaos
   metadata:
     name: payments-latency
     namespace: shoplite-staging
   spec:
     action: delay
     mode: all
     selector:
       namespaces: [shoplite-staging]
       labelSelectors: { app: checkout }
     direction: to
     target:
       mode: all
       selector:
         namespaces: [shoplite-staging]
         labelSelectors: { app: payments }
     delay: { latency: 800ms, jitter: 100ms }
     duration: 5m

4. 4

   Experiment 3: an AZ failure with AWS FIS

   The FIS AZ availability scenario disrupts networking for one AZ's subnets and can stop instances and fail over databases there. The STOP CONDITION is a CloudWatch alarm on the checkout SLO: if it fires, FIS halts and reverts. Run in staging first.

   terminalcopy cmd

   $ aws fis start-experiment --experiment-template-id EXT1a2b3c4d5e6f --tags purpose=az-gameday

   aws fis get-experiment --id EXP9z8y7x6w5v --query 'experiment.state'

   ── expected output ──

   { "status": "running", "reason": "Experiment is running." }

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.