PHASE 6Intermediate ~22 min· topic 7 of 11

Topic 6.7

Rate Limits

In one line

Providers cap how many requests and tokens you may send per minute and answer with HTTP 429 when you go over. Headers tell you how much quota is left and when it resets, and a token bucket lets you stay under the limit before the server has to refuse you.

Think of it like this

A water tank with a small tap at the bottom that drips in new water at a steady speed. You can draw a bucketful at once, and then you must wait for the tank to refill. A rate limit is the same: a burst is allowed up to the size of the tank, and the long-run speed is the speed of the drip.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Rate limit: A cap on how much you may send in a time window.
RPM / TPM: Requests per minute / tokens per minute.
HTTP 429: "Too Many Requests": you went over a limit. Try again later.
Retry-After: A header saying how many seconds to wait before trying again.
Token bucket: A counter that fills at a steady rate up to a maximum. Each request takes one token.
Burst: Many requests at once, allowed up to the bucket size.
Header: A named piece of information sent with an HTTP request or reply.

Step by step

01Step 1: meet the limit

Your batch job sends 100 requests as fast as it can. The first ones work, then the answers become 429. The service is not broken: you are being limited. The question is how to behave.

02Step 2: read what the service tells you

Run the lab below and open Raw HTTP. Find the rate-limit headers on the reply and note x-ratelimit-remaining-requests. Send again and watch it drop. The headers describe your quota right now and when it comes back.

03Step 3: turn headers into a wait

A small function reads the headers and returns milliseconds to wait: retry-after if present, otherwise the reset time of whichever quota is short. A small parser turns 1m26.4s into 86,400 ms, using only plain string scanning.

main.pywhole filepython

if int(headers["x-ratelimit-remaining-requests"]) == 0:
    wait = max(wait, parse_duration_ms(headers["x-ratelimit-reset-requests"]))
if int(headers["x-ratelimit-remaining-tokens"]) < tokens_needed:
    wait = max(wait, parse_duration_ms(headers["x-ratelimit-reset-tokens"]))

04Step 4: build the token bucket

A bucket with a capacity and a refill rate. "Try to take" tops up the bucket for the time that has passed, then either takes one token or reports how many milliseconds until one is back.

Step 4: build the token bucketdiagram

Rendering diagram…

05Step 5: use it on the client

Instead of being refused, the client asks the bucket when it may send. Eight requests wanted at once leave two immediately (the bucket holds two) and the rest start every half second, which is exactly the refill rate. The server never sees a burst it must refuse.

06Step 6: combine with retries

A limiter makes 429s rare, not impossible: other programs may share the same key, and limits can change. So keep retries with backoff and jitter for the rest (Topic 6.8) and obey Retry-After when it comes.

Try it yourself

1
Predict the denied times
For the server-side bucket, predict the lines for t=600 and t=1100 before running. (Hint: 200 milli-tokens at t=600; one half-second of drip is 1000.)
2
Change the shape
Set the client bucket to capacity 1 and refill 4 per second. Predict the eight start times, then run. Which setting controls the burst and which the long-run speed?
3
Use the lab
Send the lab request three times quickly and write down the remaining-requests value each time. How many requests per minute does this free tier seem to allow? Compare the reset time with what you expected.

Code & diagrams

A token bucket: refusing on the server, spacing on the client New tab

The clock is a number of milliseconds you pass in. Tokens are stored times 1000 so every step is a whole number.

Expected output

server side (4 tokens, 2 more every second):
  t=0: allowed
  t=0: allowed
  t=0: allowed
  t=0: allowed
  t=0: denied (429), retry in 500 ms
  t=0: denied (429), retry in 500 ms
  t=500: allowed
  t=600: denied (429), retry in 400 ms
  t=1100: allowed
  t=5000: allowed
  t=5000: allowed
  t=5000: allowed
  t=5000: allowed
  t=5000: denied (429), retry in 500 ms
client side (8 requests wanted at t=0):
  start times: 0, 0, 500, 1000, 1500, 2000, 2500, 3000

Reading rate-limit headers New tab

The header names are the ones Groq's free tier sent on 2026-10-10. The numbers in the example headers are made up. Other providers use different names.

Expected output

plenty left: wait 0 ms
request quota used up: wait 86400 ms
token quota too low for this call: wait 2085 ms
a 429 with retry-after: wait 30000 ms
parse check: 250 179560

See the rate-limit headers of a real servicereal model

Try: Press Send, then open Raw HTTP and find the headers that start with x-ratelimit. Note remaining-requests and remaining-tokens, send again, and see how they change. Which one dropped more, and why?

System

user

Modelsmall, fast, reasons firstTemperature · 0.0Max tokens (1–1024)Reasoning effort Stream (SSE) JSON mode

Goals

The reply finished on its own (finish_reason is stop)
The reply is short (12 words or fewer)

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget to move the clock on a refusal

In try_take, remove the cap: replace min(self.capacity * 1000, self.milli + ...) with just self.milli + (now - self.last) * self.rate (in every language). Run it.

terminal

$ Run

── what you'll see ──

server side (4 tokens, 2 more every second):

t=0: allowed

t=0: denied (429), retry in 500 ms

t=500: allowed

t=600: denied (429), retry in 400 ms

t=1100: allowed

t=5000: allowed

client side (8 requests wanted at t=0):

start times: 0, 0, 500, 1000, 1500, 2000, 2500, 3000

Myth vs fact

Myth

A 429 means I should retry straight away.

Fact

It means slow down. Wait at least as long as retry-after says, or longer with backoff and jitter (Topic 6.8).

Myth

Requests per minute is the only limit.

Fact

Token limits often bite first with long prompts. Read both.

Myth

Rate limits are the same for every model.

Fact

Limits are set per model and per plan, and they change. Read them from your provider's account page and headers, never from memory.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Count tokens before sending (an estimate of the input plus max_tokens) and take that many from a token-based bucket. A request-based bucket alone will not protect a TPM limit.
▸
With many servers sharing one key, each server's bucket must be a share of the limit, or the buckets must live in a shared store. Otherwise ten servers each send the full rate.
▸
Use the remaining headers as truth and your bucket as a prediction: when they disagree, slow down to match the header.
▸
Prioritise: interactive users should not wait behind a batch job. Use separate buckets or keys for batch and interactive traffic.

Remember this

1
Why this exists. A shared service must protect itself from one customer using all the capacity, whether by mistake or on purpose. So it sets limits: requests per minute (RPM) and tokens per minute (TPM), often also per day. Go over and the answer is HTTP 429 Too Many Requests. Limits depend on provider, plan and model, and free tiers have small ones.
2
The two kinds of limit. Requests per minute counts calls however small. Tokens per minute counts the size of your input plus (for many services) the output you asked for. A few huge prompts can hit the token limit while the request count is low; many tiny calls can hit the request limit while tokens are low. You must watch both.
3
Headers tell you the truth. Many services send rate-limit headers with every reply. The free tier of Groq, used by this course's labs, sent these on 2026-10-10: x-ratelimit-limit-requests, x-ratelimit-limit-tokens, x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, x-ratelimit-reset-requests (for example 1m26.4s) and x-ratelimit-reset-tokens (for example 2.085s), plus retry-after on a 429. Other providers use similar but not identical names, so read their documentation. The first lab below lets you see these in Raw HTTP.
4
Obey Retry-After. On a 429 the service often says exactly how long to wait with a retry-after header (in seconds). Waiting less just produces more 429s. The second example turns headers into a wait time: it reads the reset times, picks the longer when both quotas are short, and puts retry-after first.
5
Stay under the limit before being told off: the token bucket. A token bucket holds up to N tokens (the burst size). These are permits, not the language-model tokens of Topic 1.1; the name is just shared. Each request takes one. Tokens drip back at a steady rate. If the bucket is empty, the request waits (client side) or is refused (server side). The first example runs both sides: the server side refuses a burst of six against a bucket of four and says how long to wait, and the client side spaces eight wanted requests so none is refused.
6
Whole numbers on a simulated clock. The bucket stores its tokens times 1000 ("milli-tokens") and the clock is a number of milliseconds, so there are no floating-point surprises and every language prints the same. A real limiter reads a real clock; the logic is the same. This is also the idea behind many API gateways.

Explain it without notes

What is a rate limit and what does the server answer when you pass it?

How does a token bucket allow bursts yet keep a long-run rate?

What should a client do when it gets a 429 with a retry-after header?

Why can a limit on tokens per minute be hit with few requests?

Practice

Using the TokenBucket idea, write a function simulate(capacity, per_sec, times) that returns a list of "ok" or "wait N" strings for requests at those millisecond times, using tokens times 1000. Test it with capacity 1, 1 per second and times [0, 0, 1000].

Write parse_seconds(text) that turns strings like "2.5s" into whole milliseconds (just seconds, no minutes). Print it for "2.5s" and "0.25s".

Trade-offs

↔
A client-side limiter avoids 429s but adds waiting; a server-side limiter protects the service but pushes the waiting onto callers.
↔
A bigger burst allowance feels fast for users and is harder on the service.
↔
Header-driven waits follow reality but add parsing and provider-specific code; a bucket works anywhere but only predicts.

Done when you can

Done when you can explain RPM, TPM and HTTP 429.
Done when you can read rate-limit headers from a real reply and say what they mean.
Done when you can build a token bucket with a burst size and a refill rate.
Done when you can turn headers or retry-after into a wait time.
Done when you know why a client limiter and retries are both needed.

Command Palette

Words you'll meet

Step by step

01Step 1: meet the limit

02Step 2: read what the service tells you

03Step 3: turn headers into a wait

04Step 4: build the token bucket

05Step 5: use it on the client

06Step 6: combine with retries

Try it yourself

Predict the denied times

Change the shape

Use the lab

Code & diagrams

Break it on purpose

Forget to move the clock on a refusal

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can