Topic 6.8
Retries with Backoff and Jitter
In one line
Many model errors (429, 503, timeouts) go away if you try again later, but a naive retry loop can turn a small hiccup into an outage. This topic builds the full retry from first principles: why immediate retries pile up, exponential backoff, random jitter, Retry-After, retrying only what is retryable, and idempotency.
Think of it like this
A hundred people try to get through a shop door that sticks. If everyone pushes again the instant it fails, the doorway jams harder. If everyone waits twice as long after each failed push, they still arrive in a crowd, because they all started together. If each person waits a random time, they trickle through one by one and the door never jams.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Retry
- Trying a failed call again.
- Backoff
- Waiting longer between each attempt. Exponential backoff doubles the wait each time.
- Jitter
- Random variation added to a wait so that clients do not all retry together.
- Full jitter
- A random wait anywhere from 0 up to the current exponential ceiling.
- Thundering herd
- Many clients doing the same thing at the same moment, such as retrying together, and overloading a server.
- Retryable error
- A failure that may succeed on a later try, such as 429, 503 or a timeout.
- Idempotent
- Safe to repeat: doing it twice has the same result as doing it once.
- Idempotency key
- A unique id sent with a request so the server can recognise a repeat and not do the work twice.
Step by step
01Step 1: one call fails
A single client gets a 503. The simplest handling is a loop that tries again. It looks fine for one client and for one failure.
for attempt in range(5):
status = send()
if status == 200:
break
# no wait at all: five attempts in the same millisecond02Step 2: a hundred clients fail together
The service had a hiccup and a hundred clients got an error in the same moment. The simulation counts how many requests reach the server in each 100 ms slot. The server finishes 30 per slot, but if more than 60 arrive together it answers nobody. Retrying in the next slot gives the same 100 arrivals in slots 0, 1, 2, 3, 4 and 5: nobody succeeds, and every client gives up after six tries.
03Step 3: add exponential backoff
Now the wait doubles each time: 200, 400, 800 milliseconds and so on, up to a cap. Each client is more patient, so the server sees fewer requests per second overall. But look at the arrivals: still 100 in each busy slot, because every client uses the same schedule. Backoff spaces out the rounds but not the clients.
04Step 4: add full jitter
Each client now waits a random time between 0 and the ceiling (200, 400, 800…). The generator is the course's seeded one, with one state per client. The first two rounds are still a crowd (slots 0 and 1 have 100 arrivals each, because the early ceilings are small), but from slot 2 on the arrivals are 44, 28, 27, 6, 3 and so on. They are below the meltdown line, the server serves up to 30 per slot, and all 100 clients succeed.
def full_jitter(client, failures):
ceiling = min(CAP_MS, BASE_MS * 2 ** (failures - 1))
return random_below(client, ceiling + 1) # 0 .. ceiling05Step 5: a careful retry function
The second example is one function: classify the status (retry or stop), stop at the maximum attempts, wait with full jitter, and never wait less than Retry-After says. Scenario 2 shows the 2-second Retry-After beating a small random wait, and scenario 3 shows a 400 stopping at once.
06Step 6: make the retry safe
If the reply is lost after the server acted, the client cannot know, and a retry repeats the action. The third example sends an order twice, once without and once with an idempotency key. Without the key the shop creates two orders; with it, the second request returns the first order's id.
07Step 7: the libraries
Each library packages some of these steps. The code blocks below show tenacity, p-retry and resilience4j. Compare each with what you built: does it jitter, does it read Retry-After, how does it decide what is retryable, how does it count attempts?
Try it yourself
- 1
Predict the herd
Before running, predict how many clients succeed with "retry in the next slot". (The server is overwhelmed by 100 arrivals in every slot.) Then run and check.
- 2
Make the server stronger
Raise
MELTDOWNto 120 so 100 arrivals no longer melt the server. What happens to the first strategy now? Does backoff still matter? - 3
Change the cap
Set
CAP_MSto 800 in the herd example. Does full jitter still spread the clients out? What happens to the number that give up? - 4
Retry-After wins
In the careful retry function, change the 429 scenario's
Retry-Afterfrom 2 to 0 (or remove it). How does the printed wait change, and why?
Code & diagrams
A simulated server and 100 clients on a simulated clock. The capacity, meltdown line and delays are made-up numbers chosen to show the shapes.
Expected output
retry in the next slot
first slots (slot:arrivals): 0:100 1:100 2:100 3:100 4:100 5:100
busiest slot 100, succeeded 0, gave up 100
exponential backoff, no jitter
first slots (slot:arrivals): 0:100 2:100 6:100 14:100 30:100 62:100
busiest slot 100, succeeded 0, gave up 100
exponential backoff with full jitter
first slots (slot:arrivals): 0:100 1:100 2:44 3:28 4:27 5:6 6:3 8:4
busiest slot 100, succeeded 100, gave up 0Full jitter, Retry-After, and a list of retryable statuses. Status 0 stands for a timeout with no reply.
Expected output
503, 503, then ok: ok after 3 attempt(s), waited 977 ms
429 with Retry-After: 2: ok after 2 attempt(s), waited 2000 ms
400 bad request: stopped at once: status 400 will not get better by retrying
503 every time: gave up after 4 attempts (last status 503), waited 2302 msThe first reply is lost, so the client sends the request again.
Expected output
without a key: orders created 2, customer is told order-2
with an idempotency key: orders created 1, customer is told order-1Not run here. tenacity (Python), p-retry (JavaScript) and resilience4j (Java), written against their documentation as read on 2026-10-11. Check the version you install.
# pip install tenacity
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_random_exponential
class RateLimited(Exception): # raise these from your client for 429 and for 5xx/timeouts
pass
class ServerBusy(Exception):
pass
@retry(
retry=retry_if_exception_type((RateLimited, ServerBusy)), # a 400 raises something else, so it is NOT retried
wait=wait_random_exponential(multiplier=0.5, max=8), # random wait; tenacity's docs call this the "Full Jitter" algorithm, capped at 8 s
stop=stop_after_attempt(4), # the first try plus up to three retries
reraise=True, # after the last failure raise the real error, not a wrapper
)
def ask(prompt):
... # call the model here: raise RateLimited on 429 (read Retry-After yourself), ServerBusy on 5xx or timeout
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Retry everything
In call_with_retry, remove the retryable check (delete the if status not in RETRYABLE: block, in every language) so every failure is retried.
Myth vs fact
Myth
Exponential backoff is enough.
Fact
Without jitter, clients that failed together retry together. The herd stays the same shape, only slower.
Myth
Jitter just makes retries slower.
Fact
It makes the average wait a bit shorter or longer, and it spreads load so that more clients succeed sooner. The simulation shows the difference.
Myth
Retrying is always safe.
Fact
Only for repeatable requests. A repeat of something that changes the world (an order, a payment) needs an idempotency key.
Myth
The SDK already retries, so I'm done.
Fact
SDKs often retry a few times with their own defaults. Know them, set the numbers on purpose, and avoid stacking your retry on top of theirs (three retries around three retries is nine).
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Retries multiply load during an outage. If each of three layers retries three times, one user action becomes 27 calls. Retry in one place, usually the outermost layer that knows what is safe to repeat.
- ▸
Use a retry budget: allow retries only up to a percentage (say 10 percent) of normal traffic, so when the provider is down you don't send four times the load. Pair it with a circuit breaker (Topic 6.9).
- ▸
Set a total deadline for the whole operation, not only a count of attempts. A user waiting 40 seconds for a retry chain is worse than a clear error at 10.
- ▸
Retry-Afteris either a number of seconds or an HTTP date, and a misbehaving server can send a huge value. Parse both forms, and cap what you will accept at your total deadline: if the server asks for longer than you can wait, fail fast or fall back (Topic 6.9) instead of sleeping. - ▸
Log every retry with status, attempt number and the wait. A rising retry rate is the earliest sign of a provider problem (Topic 11.4).
- ▸
Streaming calls that break midway need extra care: the user may have seen part of the answer (Topic 5.7).
- ▸
The official OpenAI SDKs retry by default (the Python README says up to 2 times, with exponential backoff, on connection errors, 408, 409, 429 and 5xx). If you wrap them in your own retry, set
max_retries=0(Python) ormaxRetries: 0(JavaScript), as Topic 6.11 does, or the attempts multiply.
Remember this
- 1
Why this exists. Calls to a model API fail for reasons that pass: a rate limit (429), an overloaded server (503), a dropped connection, a timeout. A single retry rescues most of them, which is why Topic 0.4 added one. But how you retry matters as much as whether you do. A bad retry loop is one of the most common ways a small provider hiccup becomes a long outage of your own product.
- 2
Flaw 1: retrying at once. If a call failed because the service is overloaded, calling again right away adds more load to an already overloaded service. And if many clients do it together, they all come back at the same moment. This is the thundering herd. The first simulation sends 100 clients at a server. Time is cut into slots of 100 ms. The server finishes 30 requests per slot, and answers nobody when more than 60 arrive in one slot. Retrying in the next slot sends the same 100 again, and again.
- 3
Flaw 2: backoff alone does not break the herd. Exponential backoff doubles the wait after each failure (200 ms, 400, 800, 1600…), which gives a struggling server time to recover and slows a single client down sensibly. But clients that failed together also wait the same time, so they come back together, now at wider and wider gaps. The simulation shows 100 arrivals in every slot even with backoff.
- 4
The fix: jitter. Jitter means adding randomness to the wait. Full jitter picks a random wait between 0 and the exponential ceiling, so the herd spreads out across time and each slot sees only a few clients. The simulation prints the arrivals per slot: the first two rounds still arrive together (100 and 100), but then the arrivals spread out as 44, 28, 27, 6, 3 and so on, below the meltdown line, and all 100 clients succeed. The random numbers come from the course's seeded generator, so the output is the same every run.
- 5
Respect Retry-After, and retry only what is retryable. When the server says how long to wait (
Retry-After, Topic 6.7), never come back earlier: take the larger of your jittered wait and what the server asked. Retry only errors that can succeed later: timeouts, 408, 429, and 5xx. A 400 (your request is wrong), 401 or 403 (key or permission) or 404 will fail the same way every time; retrying wastes time and money and hides your bug. Always cap the attempts and the total time. - 6
Idempotency: retrying must be safe. If a timeout happens after the server did the work (created an order, charged a card), a retry does it twice. An idempotent request gives the same result however many times it is sent. For requests that change something, send an idempotency key: a unique id the server remembers. A repeated key returns the first result instead of repeating the action. Plain model calls (reads that generate text) are safe to repeat, but a tool that places an order is not (Phase 7).
- 7
Then use a library. Once you understand these pieces, libraries stand for them: tenacity (Python), resilience4j (Java), p-retry (JavaScript). Check what each does and does not do. For example, p-retry adds no randomness unless you turn on
randomize(and even then it is not full jitter), and the documentation of all three says nothing aboutRetry-After, so assume you must read that header yourself. Know what yours does before you rely on it. Topic 6.9 adds fallbacks and circuit breakers; Topic 6.11 puts it all together.
Explain it without notes
Explain the thundering herd and why retrying immediately makes an overload worse.
Why is exponential backoff without jitter not enough, and what does full jitter change?
Which errors should you retry and which not? What is the role of Retry-After?
What is an idempotency key and when do you need one?
Compare what tenacity, p-retry and resilience4j give you with what you built by hand.
Practice
Write is_retryable(status) returning True for 0 (timeout), 408, 429 and 500 to 599, else False. Print it for 0, 200, 400, 429, 503.
Write full_jitter_ms(attempt, base_ms, cap_ms, rand) where rand is a number from 0 up to but not including 1. Return int(rand * (ceiling + 1)) with ceiling = min(cap_ms, base_ms * 2 ** (attempt - 1)). Print the result for attempts 1 to 4 with base 500, cap 8000 and rand = 0.5.
Trade-offs
- ↔
More attempts raise the success rate but add latency and load. Cap them and set a total deadline.
- ↔
Jitter smooths load but makes any single client's wait less predictable.
- ↔
A library saves code and hides the details: know its defaults before you trust it.
Done when you can
Done when you can show how naive retries and plain backoff both leave a thundering herd.
Done when you can write full jitter with the seeded generator.
Done when you can list which statuses to retry and which not.
Done when your retry waits at least as long as Retry-After.
Done when you can explain an idempotency key and when a request needs one.
Done when you can configure tenacity, p-retry or resilience4j and say what each does not do.