Backoff on a fake 429, then a fallback
The recorded sleeps are never actually slept, so the example is instant and repeatable.
Models read and write tokens, not characters or words. A context window caps how many tokens fit in one call, cost is a simple formula of input and output tokens times rates, and rate limits mean you must trim history and retry with backoff.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write estimate_cost(input_tokens, output_tokens, input_rate, output_rate) for rates quoted per million tokens, rounded to 6 decimals.
Write backoff_delays(retries, base, cap) returning the list of waits (no jitter) for each retry.
Write first_that_works(models, prompt) that tries (name, function) pairs in order and returns the first success with its name, or raises a clear error if all fail.
Explain it without notes
What is a token, and why is counting characters a poor way to budget?
Explain the cost of a model call, and why a long chat gets expensive.
How should a client react to HTTP 429, and what should it not retry?
You have a 10-turn chat and the provider quotes per-million-token rates. How would you estimate its total cost, and name two ways to cut it?
Expected output
attempt 1 failed (primary: 429 too many requests); waiting 0.07s
attempt 2 failed (primary: 429 too many requests); waiting 0.85s
[primary] answer to 'hi'
sleeps recorded, not slept: 2
attempt 1 failed (big: 429 too many requests); waiting 0.38s
fallback: moving to the next model
[small] answer to 'hi'