Topic 10.14
Retries
In one line
The last request failed. When do you retry, how hard, and how do retries stop being the retry-storm?
Think of it like this
Calling a friend who didn't pick up: you don't call back instantly ten times in a row (that's rude and useless); you wait a bit longer between each attempt (backoff) so you're not just spamming a busy line.
Key ideas
- 01
Idempotency is what makes retries safe: retry against a non-idempotent POST without a key = duplicates (Phase 4 payment).
- 02
Backoff: fixed → jittered exponential (base × 2^n + random) — jitter breaks synchronized retry storms.
- 03
Retry policy: 3 attempts with backoff to the live queue, then → DLQ (dead letter) so human/alerting can intervene.
- 04
Never retry forever: cap attempts + time budget; a stuck retry loop is a slow outage.
- 05
Retry storms: 10k failures all retry → burst; jitter + circuit breakers (10.16/11.8) flatten it.
- 06
Interview: 'idempotent handler, exponential backoff with jitter, cap at 3, then dead letter'.
Java / Spring map
- →
Spring Retry (@Retryable(backoff = @Backoff(delay=100, multiplier=2, jitter=50))), Resilience4j Retry.
Explain without notes
Why does a fleet retrying on the same schedule amplify an outage? What two mechanisms break the loop?
Practice
Write the retry + DLQ configuration for an SMS sender that fails 5% of the time.
Trade-offs
- ↔
Aggressive retries = resilience vs thundering herd; backoff dial is the compromise. Cap it and DLQ it.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
My default policy is: idempotent + jittered exponential backoff + cap + DLQ — every time.