Idempotency Keys, Timeouts & Safe Retries

In one line

Networks fail after the server did the work but before the client heard back. Idempotency keys let clients retry 'create payment' safely; timeouts and retry rules keep one slow dependency from taking everything down.

0/7 · 0%

Think of it like this

Pressing a lift button. Pressing it five times still brings one lift: the action is idempotent. A 'pay ₹500' button is not: pressing twice could charge twice, unless the shop recognises that it's the same purchase.

Key ideas

01
The problem: the client sends POST /payments, the server charges the card, and then the connection drops. The client doesn't know if it worked. Retrying might double-charge; not retrying might lose the order.
02
IDEMPOTENCY KEY: the client generates a unique key per logical operation (a UUID) and sends it in a header (Idempotency-Key: 5f1c…). The server stores key → result (in the same transaction as the effect, or in Redis/DB with a unique constraint). A repeated key returns the STORED result without redoing the work. Stripe popularised this pattern; payment APIs require it.
03
TIMEOUTS on every outbound call, shorter than the caller's own deadline. RETRIES only for idempotent operations or with idempotency keys, with EXPONENTIAL BACKOFF and JITTER (random spread) so thousands of clients don't retry in sync, and a cap (e.g. 3 attempts, or a retry budget). Retry transient errors (timeouts, 503, 429 after Retry-After), never 400-class validation errors.
04
This is the API side of the same idea as idempotent consumers in messaging (Phase 10, idempotency) and circuit breakers (Phase 11).

Java / Spring map

→
Resilience4j's Retry (with IntervalFunction.ofExponentialRandomBackoff) and TimeLimiter wrap client calls; an idempotency filter can store (key, response) in a table with a unique index on the key.

Code & diagrams

IdempotentPayments.javajava

Store the result under the client's key; a retry returns the stored result instead of charging again.

@PostMapping("/payments")
ResponseEntity<Payment> pay(@RequestHeader("Idempotency-Key") String key, @RequestBody PayRequest req) {
  return idempotency.findByKey(key)
      .map(saved -> ResponseEntity.ok(saved.payment()))                 // replay: same answer, no new charge
      .orElseGet(() -> {
        Payment p = payments.charge(req);                                // do the work once
        idempotency.save(new IdempotencyRecord(key, req.hash(), p));     // unique(key) guards races
        return ResponseEntity.status(HttpStatus.CREATED).body(p);
      });
}

Explain without notes

Why add jitter to retry backoff?

Practice

Two identical requests with the same idempotency key arrive at the same time on different servers. How do you avoid charging twice?

Trade-offs

↔
Idempotency storage adds a write per request and a retention policy (e.g. 24 h) but makes retries safe; without it, you must choose between possible duplicates and possible loss.

Run it in production

You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:

Stateful · Consumer groups & delivery semantics

Rebalancing, offsets, at-least-once, idempotency, DLQs, and consumer lag.

Observability · Retry storm

What unbounded retries look like in metrics and traces.

Completion checklist

I can explain why retries without idempotency are dangerous
I set timeouts, backoff with jitter, and retry limits on outbound calls

Back to phase

Command Palette