A job queue with retries and idempotency keys
One worker and a simulated clock in milliseconds. The fake model times out on purpose.
Some LLM work takes too long for a web request: summarising a long document, processing a batch, running an agent. Put such work in a queue, return a job id at once, let a worker do it with retries and backoff, and use idempotency keys so repeated submissions do not repeat the work.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write backoff_ms(attempt, base=1000, cap=8000) for the exponential schedule 1000, 2000, 4000, 8000, 8000 (no jitter) and print it for attempts 1 to 5.
Write submit_once(store, key, payload) using a dict store that returns the same job id for the same key. Show three submissions with two distinct keys.
Explain it without notes
When should an LLM task become a background job instead of a request?
What does an idempotency key do, and where is it stored?
Why is at-least-once delivery a problem for LLM jobs, and how do you handle it?
How would you retry failed jobs safely?
Expected output
submit k1: job-1 (new)
submit k2: job-2 (new)
submit k1: job-1 (duplicate, same job)
submit k3: job-3 (new)
t=0 job-1 attempt 1: succeeded
t=500 job-2 attempt 1: timeout, retry at t=1500
t=1000 job-3 attempt 1: timeout, retry at t=2000
t=1500 job-2 attempt 2: succeeded
t=2000 job-3 attempt 2: timeout, retry at t=4000
t=4000 job-3 attempt 3: failed for good
job-1 succeeded attempts=1 result=summary of report
job-2 succeeded attempts=2 result=summary of flaky
job-3 failed attempts=3 result=-