Topic 12.4
Queues and Background Jobs
In one line
Some LLM work takes too long for a web request: summarising a long document, processing a batch, running an agent. Put such work in a queue, return a job id at once, let a worker do it with retries and backoff, and use idempotency keys so repeated submissions do not repeat the work.
Think of it like this
Dropping clothes at a dry cleaner. You do not stand at the counter for two days. You hand them over, get a ticket with a number, and come back later. If you accidentally hand in the same bag twice, a good shop notices the ticket and does not clean it twice. The ticket is the job id, the back room is the worker, and the check for the same bag is the idempotency key.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Job
- A stored record of one piece of work: its input, status, attempts and result.
- Queue
- A waiting line of jobs; workers take from the front.
- Worker
- A process that takes jobs from the queue and runs them.
- Polling
- The client asking again and again, "is my job done yet?"
- Backoff
- Waiting longer after each failed attempt before trying again.
- Idempotency key
- A unique label per intended action; the same key again returns the first result instead of repeating the work.
- At-least-once
- A promise that a job runs one or more times, never zero, so it may run twice after a crash.
- Visibility timeout
- The time after which a job that has been running too long is treated as stuck and put back in the queue.
- Webhook
- A web address you give the service so it can call you when something is finished.
- Dead-letter list
- A place where jobs that failed for good are kept for a person to look at.
Step by step
01Step 1: the request that took too long
A summary of a long document takes 40 seconds when the provider is slow. The user waits, a proxy cuts the connection at 30 seconds, the user presses the button again, and now two summaries are being paid for. Three problems: waiting, cutting and repeating.
02Step 2: answer fast, work later
The endpoint stores a job and answers 202 with {"job_id": "job-1"} in milliseconds. The client polls the job. Nothing about the model call is in the request any more.
job_id, created = queue.submit(key, payload)
status = 202 if created else 200
return status, {"job_id": job_id}03Step 3: the lifecycle
Every job follows the states in the picture. The only way back is from running to queued after a failure with attempts left, and the delay before the next try grows with each attempt.
04Step 4: run the simulation
The example has one worker and a simulated clock that moves in 500 ms steps. flaky times out on its first attempt only, broken always times out. Watch job-2 time out at t=500, wait 1000 ms and succeed at t=1500. job-3 (the broken one) uses all three attempts, with waits of 1000 and 2000 ms, and fails for good at t=4000. The wait here is 1000 x attempt, a simple growing step so the output is easy to check. Real systems usually double it (1000, 2000, 4000...), as the practice below does.
05Step 5: submit the same key twice
The third submission reuses key k1. The queue returns job-1 and creates nothing. This is what protects you from double clicks and client retries. Without it (see Break it) the same report is generated and paid for twice.
06Step 6: what a real system adds
A durable store for jobs (a database table or a queue service) instead of a list, several workers taking jobs safely (for example with SELECT ... FOR UPDATE SKIP LOCKED in PostgreSQL, or the queue's own lease), a visibility timeout, jitter, a dead-letter list of failed jobs for a human to look at, and metrics for queue length, age of the oldest job and failure rate (Topic 11.4).
Try it yourself
- 1
Predict the timeline
Before running, write when
job-2andjob-3are tried (job-1 finishes at t=0). Remember a worker takes one job per tick, and the retry time isnow + 1000 * attempts, checked every 500 ms. - 2
Raise the attempts
Set
MAX_ATTEMPTSto 4. When doesjob-3finally fail (the waits are 1000, 2000, then 3000 ms), and does the simulation end (the clock stops at 8000) before it does? - 3
Add a second worker
Change
tickso that it can run up to two due jobs per tick. What happens to the timestamps ofjob-1andjob-2?
Code & diagrams
One worker and a simulated clock in milliseconds. The fake model times out on purpose.
Expected output
submit k1: job-1 (new)
submit k2: job-2 (new)
submit k1: job-1 (duplicate, same job)
submit k3: job-3 (new)
t=0 job-1 attempt 1: succeeded
t=500 job-2 attempt 1: timeout, retry at t=1500
t=1000 job-3 attempt 1: timeout, retry at t=2000
t=1500 job-2 attempt 2: succeeded
t=2000 job-3 attempt 2: timeout, retry at t=4000
t=4000 job-3 attempt 3: failed for good
job-1 succeeded attempts=1 result=summary of report
job-2 succeeded attempts=2 result=summary of flaky
job-3 failed attempts=3 result=-Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the idempotency check
In submit, skip the lookup: replace if key in self.by_key: with if False: (JavaScript: if (false), Java: remove the containsKey line).
Myth vs fact
Myth
A bigger timeout is the fix for slow generations.
Fact
The user's browser, the proxy and the platform each have their own limit. A queue removes the dependency on the request staying open.
Myth
Retrying a failed job is always safe.
Fact
Only if the job can repeat without harm. Use idempotency keys and check for finished work before the expensive or side-effecting step.
Myth
Exactly-once processing is a setting you can turn on.
Fact
Most systems give at-least-once. You get the effect of exactly-once by making work idempotent.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Add jitter to backoff (a random part of the wait, Topic 6.8) so a provider outage does not turn into a synchronized wave of retries when it ends.
- ▸
Record, per job, the model, prompt version, tokens and cost (Topic 11.4). A queue is also where you attribute cost to tenants (Topic 12.5).
- ▸
Watch the age of the oldest queued job, not just the queue length: it tells you how long a customer waits, and is the best alert for stuck workers.
- ▸
Progress UIs: store a coarse progress value or partial results on the job so the poll can show them (Topic 5.6 for partial output).
Remember this
- 1
Why this exists. An HTTP request that waits for a model for 30 seconds, or for minutes with tools and retries, is fragile. Browsers and proxies cut long requests, serverless platforms have time limits (Topic 12.8), a deploy kills the process in the middle, and the user may close the tab. Moving the work out of the request makes it survive all of that.
- 2
The shape. The endpoint validates the request, stores a job row with status
queued, and answers immediately with202 Acceptedand the job id. A worker (a separate process or thread) takes queued jobs, runs them and writes the result. The client pollsGET /v1/jobs/<id>(simple and robust), or receives a webhook (your service calls a web address the client gave you when the job is done) or a push event. See the System Design course (/system-design) for queue technologies. - 3
The lifecycle. A job moves through a few states:
queued,running,succeeded,failed. Store the attempt count, the last error and timestamps. Write each transition to storage, so a crash leaves a clear record. A job that has beenrunningfor much longer than the maximum possible time is stuck: a visibility timeout puts it back in the queue. - 4
Retries with backoff. Provider timeouts and 429s are normal (Topic 6.8). The worker retries a failed attempt after a delay that grows with each attempt, up to a maximum number of attempts, then marks the job
failedwith the reason. Recall from Topic 6.8: retry only retryable errors, because a 400 will never succeed. Add jitter in real systems so many retried jobs do not wake at the same moment (the example uses plain steps so the output is exact). Respect the provider'sRetry-After. - 5
Idempotency. Networks repeat requests: the client times out and tries again, a user double-clicks, a webhook is delivered twice. If every repeat creates a new job, you do the work and pay for it twice. The caller sends an idempotency key (a unique string per intended action); the service stores key to job id and returns the existing job when the key appears again. Also make the job's own side effects safe to repeat (use the job id as a key for the email or database write it makes).
- 6
At-least-once delivery. Most queues promise that a job runs at least once, not exactly once: after a crash it may run again. So the worker's work must be idempotent, or it must check "did I already finish this?" before the expensive step. Design for it from the start.
- 7
Limits and priorities. A queue can grow without bound when jobs arrive faster than workers finish. Cap the queue length per tenant (Topic 12.5), return 429 when full, and keep slow batch work apart from interactive requests so a nightly import cannot delay a person waiting on a reply. For offline bulk work also look at the provider's batch APIs (Topic 6.10).
Explain it without notes
When should an LLM task become a background job instead of a request?
What does an idempotency key do, and where is it stored?
Why is at-least-once delivery a problem for LLM jobs, and how do you handle it?
How would you retry failed jobs safely?
Practice
Write backoff_ms(attempt, base=1000, cap=8000) for the exponential schedule 1000, 2000, 4000, 8000, 8000 (no jitter) and print it for attempts 1 to 5.
Write submit_once(store, key, payload) using a dict store that returns the same job id for the same key. Show three submissions with two distinct keys.
Trade-offs
- ↔
Queues make long work reliable and add moving parts: storage, workers, monitoring.
- ↔
Polling is simple and robust; webhooks and push are faster but need more infrastructure and retry logic of their own.
- ↔
More attempts raise the success rate and the cost of failures; set a cost-aware cap.
Done when you can
Done when you can explain why a slow LLM call belongs in a job.
Done when you can draw the job states and the transitions between them.
Done when you can implement retries with growing waits and a maximum number of attempts.
Done when you can explain idempotency keys and at-least-once delivery.
Done when you can name the metrics that show a queue is unhealthy.