PHASE 15Intermediate ~29 min· topic 2 of 10

Topic 15.2

Robust API Clients: Retries, Backoff and Rate Limits

In one line

Networks and servers fail in short bursts, so a good client retries only the failures that can pass (timeouts, 429, 5xx), waits longer after each try with random jitter, honours the server's Retry-After header, and never repeats a request that is not safe to repeat.

Think of it like this

Calling a busy restaurant to book a table. (Recall from Topic 15.1: a status code is the number on the response slip, and 4xx versus 5xx tells you whose fault it is.) The line is engaged. If you redial every half second, you are one of a hundred people hammering the same phone, and the line stays busy. A sensible caller waits a bit, then a bit longer, and not at exactly the same moment as everyone else. If the person who answers says "call back in ten minutes", you do exactly that. This is retry with backoff and jitter, and Retry-After.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Transient error: A failure that goes away if you wait and try again, such as a timeout or a 503.
Permanent error: A failure that will repeat every time, such as a 404 or a malformed request.
Backoff: Waiting longer between each retry. Exponential backoff doubles the wait each time.
Jitter: Random variation added to a wait so many clients do not retry at the same instant.
Rate limit: A cap on how many requests (or tokens) a client may use per period.
429: The status code for Too Many Requests, meaning you hit a rate limit.
Retry-After: A response header telling you how long to wait before trying again.
Idempotent: Doing it twice has the same effect as doing it once, so retrying is safe.
Idempotency key: A unique id you send so the server can spot a repeated request and not do the work twice.
Pagination: Splitting a long list into pages that you fetch one after another.

Step by step

01Decide: retry or give up?

The first question after any failure is whether waiting can help. Timeouts and connection errors can. Status codes 429, 500, 502, 503 and 504 usually can. Everything else in 4xx cannot.

Decide: retry or give up?diagram

Rendering diagram…

02A fake flaky server

To test retries you need a server that fails on purpose. This one counts calls and answers 503 for the first two, then succeeds. Real flaky servers do exactly this.

main.pywhole filepython

calls = {"n": 0}


def flaky(request):
    calls["n"] += 1
    if calls["n"] <= 2:
        return httpx.Response(503, json={"error": "overloaded"})
    return httpx.Response(200, json={"answer": 42})

03Backoff with full jitter

After attempt number n (counting from 1) the ceiling is base * 2 ** (n - 1), capped. The actual wait is a random number between 0 and the ceiling. A seeded random.Random(7) makes the example repeatable, and a sleep function passed in as a parameter lets us skip real waiting in tests.

Passing sleep and rng in is the key design trick: production passes time.sleep and a normal random generator, tests pass a recorder and a seeded one. Recall from Topic 5.6: a function is a value, so you can hand it to another function as a parameter.

random.uniform(0, 3) gives a random decimal number between 0 and 3 (Topic 6.4). A seeded generator always produces the same "random" numbers, so the output is repeatable.

main.pywhole filepython

ceiling = min(cap, base * 2 ** (attempt - 1))
delay = rng.uniform(0, ceiling)
sleep(delay)

04Honour Retry-After

When a 429 or 503 carries Retry-After: 3, the server is telling you the earliest useful time. Read it with float(response.headers["retry-after"]) and wait at least that long. The header can also hold a date instead of seconds; production code should handle both forms (the standard library parses dates with email.utils.parsedate_to_datetime).

05Paginate with a cursor

A cursor is an opaque token for "where I stopped". Send it back as a query parameter to get the next page, and stop when the server returns no next cursor. Collect items as you go.

Opaque means you should not look inside or build it yourself; just pass it back exactly as received.

main.pywhole filepython

def fetch_all(client):
    items, cursor = [], None
    while True:
        params = {"cursor": cursor} if cursor else {}
        data = client.get("/items", params=params).json()
        items.extend(data["items"])
        cursor = data["next"]
        if cursor is None:
            return items

06Idempotency keys for writes

If a POST times out, you do not know whether the server did the work. Retrying blindly could duplicate it. Generate one key per *logical* operation (uuid.uuid4()), send it as a header on every attempt of that operation, and let the server deduplicate. Check each API's docs: not all of them support keys, and the header name varies.

main.pywhole filepython

import uuid

key = str(uuid.uuid4())   # made once, reused for every retry of this payment
headers = {"Idempotency-Key": key}
# response = client.post("/payments", json=payload, headers=headers)

Try it yourself

1
Make the server worse
In the first example change calls["n"] <= 2 to <= 10 and run it. Predict the result: five attempts, then RuntimeError: gave up after 5 attempts.
2
Change the seed
Change random.Random(7) to random.Random(8). The delays change but the pattern (each below its ceiling) does not. Jitter is exactly this: spread, not a fixed number.
3
Add a permanent error
Make the server return httpx.Response(404) on the first call. The helper returns the 404 response immediately: 404 is not in RETRYABLE. That is the "fail fast" half of retrying.

Code & diagrams

A retry helper with backoff, jitter and a flaky server New tab

Change Random(7) to Random(8) and see different waits. The sleep is a recorder, so nothing really waits. The `*` in the function's parameter list means everything after it must be given by name (Topic 5.4). The `else:` after `try` runs only when no error happened (Topic 7.1).

Expected output

attempt 1: status 503
  waiting 0.16s (ceiling 0.5s)
attempt 2: status 503
  waiting 0.15s (ceiling 1.0s)
final: 200 {'answer': 42}
total fake sleeping: 0.31 seconds in 2 waits

Retry-After and cursor pagination New tab

The fake server answers 429 once, then 200, and serves three pages of items.

Expected output

first: 429 Retry-After: 3
the server asks us to wait 3.0 seconds (we skip the real sleep here)
second: 200 {'ok': True}
pages: 3 items: [1, 2, 3, 4, 5, 6, 7]

How the wait grows (ceiling per attempt)diagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Retry forever with no wait

Write a retry loop with no attempt limit and no sleep against a server that keeps failing.

main.pywhole filepython

import httpx

client = httpx.Client(
    base_url="https://api.example.com",
    transport=httpx.MockTransport(lambda request: httpx.Response(503)),
)
tries = 0
while True:
    tries += 1
    if client.get("/answer").status_code == 200 or tries == 100000:
        break
print("tries:", tries)

terminal

$ python main.py

── what you'll see ──

tries: 100000

Myth vs fact

Myth

Retrying always makes a program more reliable.

Fact

Only for transient errors and safe requests. Retrying a permanent error wastes time, and retrying a non-idempotent POST can duplicate the action. Unbounded retries can also make an outage worse.

Myth

Jitter is just noise.

Fact

It is the part that stops many clients from retrying at the same instant. Backoff without jitter still produces synchronised waves.

Myth

If I get a 429 I should retry right away.

Fact

A 429 means slow down. Honour Retry-After if present, then back off; and consider limiting your own request rate in advance (for example with a semaphore, Topic 12.7).

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Libraries already exist for this: tenacity (a general retry decorator) and httpx's HTTPTransport(retries=n), which retries only *connection* failures. The official AI SDKs also retry 429 and 5xx by default with backoff. Writing the loop once yourself shows what they do and what settings (max retries, timeout) to tune.
▸
Retries multiply across layers. If three services each retry three times, one user request can become 27 calls to the bottom service. Retry at one layer, or pass a deadline down so lower layers stop when the caller has already given up.
▸
Distinguish a rate limit (a cap you can plan for: pace your requests, use a token bucket or semaphore) from overload (the server is hurt: back off). With AI APIs the limit is often tokens per minute as well as requests per minute, so a few huge prompts can trigger 429 even at low request counts.
▸
Make timeouts and retry budgets configuration, not constants buried in code (Topic 15.3). The right values differ between a quick lookup and a long model call.
▸
Only the connect phase is always safe to retry. A ReadTimeout on a POST means the request may already have been processed, so retry it only when the API accepts an idempotency key. The helper in this topic is written for GET. A production version takes the method into account: retry connection errors for any method, but retry read timeouts and 5xx only for idempotent methods or keyed requests. Also treat 408 (Request Timeout) as retryable, and keep RETRYABLE as data you can configure per API.
▸
Retry-After is either a number of seconds or an HTTP date (Wed, 21 Oct 2026 07:28:00 GMT). float(header) handles only the first form, so a robust client tries float(...) and falls back to email.utils.parsedate_to_datetime(...) minus the current UTC time. Cap the result too: a server that says "retry in one hour" should not make a request handler sleep for an hour. Fail fast or hand the job to a queue instead.
▸
Full jitter (uniform(0, ceiling)) can pick a wait close to zero, which is fine for spreading load but weak as a courtesy. Variants keep a floor: *equal jitter* waits ceiling/2 + uniform(0, ceiling/2), and *decorrelated jitter* picks uniform(base, previous_wait * 3). All three beat a fixed delay. For a service under sustained failure, add a circuit breaker: after enough consecutive failures stop calling for a cool-off period, instead of letting every request pay the full retry cost.

Remember this

1
Failures come in two kinds. A transient failure goes away if you wait: a timeout, a dropped connection, a 503 Service Unavailable, a 429 Too Many Requests. A permanent failure will repeat forever: a 400 (your request is malformed), a 401 or 403 (not allowed), a 404 (not there). Retrying a permanent failure only wastes time and may get you blocked. Retry transient ones; fail fast on permanent ones.
2
Exponential backoff doubles the wait after each failed try: 0.5 s, 1 s, 2 s, 4 s, up to a cap such as 30 s. The doubling gives a struggling server growing room to recover. Always add a maximum number of attempts so the client eventually gives up and reports the problem.
3
Jitter is random variation added to the wait. Without it, a thousand clients that failed at the same moment all retry at the same moment, and the server is hit by a wave again (a *thundering herd*). The common recipe, called full jitter, picks the wait randomly between 0 and the backoff ceiling: random.uniform(0, ceiling).
4
Rate limits are the server's way of protecting itself: so many requests per minute, or so many tokens per minute for an AI model (a token is a small chunk of text, about three quarters of a word; Topic 16.4). When you exceed one you get 429 Too Many Requests, often with a **Retry-After** header that says how many seconds to wait (or an exact date). Obey it. It is better information than your own guess, so use max(backoff, retry_after).
5
Idempotency decides whether a retry is safe. A request is *idempotent* if doing it twice has the same effect as once. GET, HEAD, OPTIONS, PUT and DELETE are meant to be (PATCH is not guaranteed to be); POST usually is not, so a retried POST /payments could charge a customer twice if the first request actually succeeded and only the reply was lost. APIs that matter solve this with an idempotency key: a unique value you send in a header (commonly Idempotency-Key), so the server can recognise a repeat and return the first result.
6
Big result lists arrive in pages. The API returns a first page plus a way to ask for the next one: a next cursor or token, a page number, or a Link header. A client loops until there is no next page. The same retry helper wraps each page request, so one bad page does not lose the earlier ones. All of this is exactly what you need to call large language model APIs, where 429 and 5xx errors are routine at busy times (Phase 16).

Explain it without notes

Which failures should a client retry, and which should it not?

Explain exponential backoff with jitter and why the jitter is needed.

What is Retry-After, and how does a client use it with its own backoff?

What is idempotency, and how does it affect retrying a POST request?

A POST request to create an order times out. What do you do?

Practice

Write backoff_delays(attempts, base, cap) that returns the list of backoff ceilings with no randomness. For (5, 0.5, 4.0) it should be [0.5, 1.0, 2.0, 4.0, 4.0].

Write is_retryable(status) that returns True for 429 and any 5xx except 501, False otherwise.

Write a function wait_time(response, backoff) that returns the larger of the backoff and the Retry-After header (in seconds, if present).

Trade-offs

↔
More retries versus faster failure. More attempts hide short outages but make users wait and add load. Pick limits from how long the caller can really wait.
↔
Writing the loop yourself versus a library. Your own loop is easy to read and test. tenacity or an SDK's built-in retries cover more cases (async, custom stop rules) but hide the policy; read their defaults before trusting them.

Done when you can

Done when you can list the transient and permanent failures of an HTTP call.
Done when you can write exponential backoff with full jitter and explain each part.
Done when you can read Retry-After and combine it with your own wait.
Done when you can say which HTTP methods are safe to retry and what an idempotency key does.
Done when you can loop through a paginated API using a cursor.
Done when you pass sleep and a random generator into the helper so it can be tested instantly.

Command Palette

Words you'll meet

Step by step

01Decide: retry or give up?

02A fake flaky server

03Backoff with full jitter

04Honour Retry-After

05Paginate with a cursor

06Idempotency keys for writes

Try it yourself

Make the server worse

Change the seed

Add a permanent error

Code & diagrams

Break it on purpose

Retry forever with no wait

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can