“Payments got slow for two minutes. Then it stayed down for forty — and got four times our normal traffic.”

1. 18:00

   FINDING

   Yesterday's 'resilience' change: retry payments up to 3 times

   let pay;
   for (let attempt = 0; attempt < 4; attempt++) {           // 1 try + 3 retries, no delay
     try {
       pay = await fetch(`${PAYMENTS_URL}/charge`, { method: "POST", /* ... */ signal: AbortSignal.timeout(1000) });
       if (pay.ok) break;
     } catch (err) {
       if (attempt === 3) throw err;
     }
   }

2. 18:22

   ALERT

   Checkout errors at 100%; payments p99 at 2.6 s

   Every checkout times out four times, then fails.
3. 18:26

   QUERY

   Payments is receiving 4× the calls, while checkouts are flat

   Compare the rate of requests INTO payments with the rate of checkouts. Normally the ratio is 1.0. Right now each checkout makes four attempts, so the dependency that was struggling now has four times the load, which is exactly what stops it recovering.

   PromQL· Prometheuscopy

   sum(rate(http_request_duration_seconds_count{job="payments", route="/charge"}[1m]))
   /
   sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[1m]))

   result

   {}  3.97

4. 18:28

   QUERY

   One checkout, four failed siblings

   The trace shows four POST spans to payments back to back, each cut off at exactly 1,000 ms by the timeout and marked as errors. Retrying immediately gives a struggling service no time to recover, and all clients retrying in lockstep make each wave of load arrive at once.

   POST /checkoutshoplite

   4012 ms

   POST (payments) attempt 1shoplite

   1000 ms

   POST /chargepayments

   2590 ms

   POST (payments) attempt 2shoplite

   1000 ms

   POST /chargepayments

   2600 ms

   POST (payments) attempt 3shoplite

   1000 ms

   POST (payments) attempt 4shoplite

   1000 ms

5. 18:31

   HYPOTHESIS

   Worse: payments kept processing the abandoned attempts

   Notice the server spans continue for 2.6 s after the client gave up. Timed-out attempts still reach the provider. Without an idempotency key, a retried charge that eventually succeeds could charge the customer TWICE.
6. 19:02

   RESOLVED

   Retries disabled via config; payments recovers within 2 minutes of the load dropping

1. 01Backoff, jitter, one retry, an idempotency key

   One retry at most, after 100–300 ms of randomised delay, and only while the overall 2-second deadline allows. The same Idempotency-Key on both attempts lets payments (and real providers like Stripe or Razorpay) return the original result instead of charging twice. Switch the scenario off afterwards.

   const idempotencyKey = crypto.randomUUID();
   const deadline = Date.now() + 2000;
   let pay;
   for (let attempt = 0; attempt < 2; attempt++) {
     const remaining = deadline - Date.now();
     if (remaining <= 0) break;
     try {
       pay = await fetch(`${PAYMENTS_URL}/charge`, {
         method: "POST",
         headers: { "content-type": "application/json", "idempotency-key": idempotencyKey },
         body: JSON.stringify({ amount: qty * 120 }),
         signal: AbortSignal.timeout(Math.min(1000, remaining)),
       });
       if (pay.ok || pay.status < 500) break;   // don't retry 4xx
     } catch (err) {
       req.log.warn({ err, attempt }, "payments attempt failed");
     }
     retries.inc();
     await new Promise((r) => setTimeout(r, 100 + Math.random() * 200)); // backoff + jitter
   }

   terminalcopy cmd

   $ curl -s -X POST localhost:8081/admin/chaos -H 'content-type: application/json' -d '{"scenario":"slow-payments","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Measure retries, and alert on amplification

   Declare const retries = new client.Counter({ name: "shoplite_payment_retries_total", help: "Payment call retries" }). Alert when retries exceed 10% of checkouts: at that point retries are adding load rather than masking rare blips.

         - alert: PaymentRetryAmplification
           expr: |
             sum(rate(shoplite_payment_retries_total[5m]))
             / sum(rate(http_request_duration_seconds_count{job="shoplite", route="/checkout"}[5m]))
             > 0.1
           for: 2m
           labels: { severity: warning, service: shoplite }
           annotations:
             summary: "Payment retries are {{ $value | humanizePercentage }} of checkouts — dependency degraded, retries amplifying load"

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.