Topic 6.10
Concurrency and Batching
In one line
Running many model calls at once finishes a big job much sooner, but too many at once hits rate limits. A concurrency limit keeps a fixed number in flight, results are stored by position so order survives, and batch APIs handle offline work more cheaply.
Think of it like this
A post office with several counters. One clerk serving 100 people takes all day. Eight clerks serve them in a fraction of the time, but if you hire a hundred clerks there is no room behind the counter and they all fight over the stamp drawer. The right number is a limit. And the customers leave in a different order than they joined, so each parcel carries a tag saying whose it is.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Concurrency
- Several things in progress at the same time.
- Semaphore
- A counter of free places. Work takes a place to start and returns it when done.
- In flight
- Started but not yet finished.
- Makespan
- The total time from the start of a job until its last piece finishes.
- Batch API
- A provider service where you submit many requests together and collect results later, often cheaper.
- custom_id
- Your own label on each batch request, returned with its result so you can match them up.
Step by step
01Step 1: one at a time is slow
Eight calls take 300, 500, 200, 400, 100, 600, 250 and 350 simulated milliseconds. One after the other they take 2,700 ms, the sum. For 5,000 calls that is the difference between minutes and hours.
02Step 2: all at once is risky
With a limit of 8, all start together and the job ends when the slowest call ends (600 ms). That is the fastest possible, and for 5,000 calls it would be 5,000 simultaneous requests: rate limits, memory, and angry neighbours on the same API key.
03Step 3: a limit in the middle
Keep at most N running. When one finishes, start the next waiting call. In the scheduler loop, running holds the calls in flight with their finish times, the clock jumps to the next finish, and a free place starts the next job.
while len(running) < limit and next_job < len(DURATIONS):
running.append((clock + DURATIONS[next_job], next_job))
next_job += 104Step 4: results by position
The finish order printed for limit 3 is not 0, 1, 2, 3…. Results are stored in a list indexed by the job's number, so the final list is in request order whatever order the calls ended in.
05Step 5: batch for offline work
The second example sends five documents as one batch. Results return in a different order, and one request failed and is missing. Matching by custom_id puts each answer back with its document, and the missing id is collected to send again.
06Step 6: put it together
A real job: pool with a limit, per-call retries with jitter, a rate limiter shared by all workers, a cost meter, and results stored by id so the job can resume after a crash. Topic 6.11 assembles the single-call version of this.
Try it yourself
- 1
Predict the time for limit 2
Add
2to the limits list and predict the finish time by hand: start jobs 0 and 1 (300 and 500), then each free place takes the next job. Run it and compare. - 2
Order of the finishing
For limit 3, write down which job finishes first and which finishes last before running. Why does the last job to finish not have to be the last job to start?
- 3
Lose a result
In the batch example, make the fake API also drop
doc-1. Does the code still print the right missing list?
Code & diagrams
A scheduler loop on a simulated clock. Real code would use asyncio or a thread pool; the logic of the limit is the same.
Expected output
limit 1: all done at 2700 ms, at most 1 at once
finished in this order: 0 1 2 3 4 5 6 7
limit 3: all done at 1000 ms, at most 3 at once
finished in this order: 2 0 4 1 3 6 7 5
limit 8: all done at 600 ms, at most 8 at once
finished in this order: 4 2 6 0 7 3 1 5
results in request order: answer-0, answer-1, answer-2, answer-3, answer-4, answer-5, answer-6, answer-7The 50 percent price and the price per token are example numbers. A batch API has its own price, delay and limits, so read your provider's terms.
Expected output
came back in this order: doc-4, doc-2, doc-1, doc-0
doc-0: ALPHA
doc-1: BRAVO
doc-2: CHARLIE
doc-3: MISSING
doc-4: ECHO
send again in the next batch: doc-3
cost if sent one by one: 20000 micro-dollars, as a batch: 10000 micro-dollarsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Append results as they finish
In run_all, replace results[job] = f"answer-{job}" with an append to a list that you create as results = [] (in the other languages use an add/push to a growing list), then print the list.
Myth vs fact
Myth
More concurrency is always faster.
Fact
Only until you hit the provider's limits or your own machine's limits. Then it gets slower, with retries and errors.
Myth
Calls that start first finish first.
Fact
Calls take different times. Results come back in any order, so tag them.
Myth
Batch APIs are just cheaper live calls.
Fact
They trade waiting for price and have their own limits, formats and failure handling.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Share one limiter (token bucket) and one concurrency limit across all workers in a process, and divide the quota between processes. Ten processes each running the full limit send ten times the allowed rate.
- ▸
Make long jobs resumable: store each result by id as it arrives, and on restart skip ids already done. A job that must start over after a crash at 90 percent wastes money.
- ▸
A quick way to size the limit is Little's law: calls in flight = calls per second × seconds per call. At 5 calls per second and 2 seconds per call you need about 10 in flight. Output-heavy calls last longer, so the same quota supports fewer parallel calls than short ones; watch tail latency (Topic 6.3), not only the average.
- ▸
Use a bounded queue between a producer and the workers so a fast producer cannot fill memory with waiting requests.
- ▸
Cancel work on shutdown: stop starting new calls, let in-flight calls finish or abort them (Topic 5.5), and save partial results.
Remember this
- 1
Why this exists. Many jobs involve lots of independent calls: classify 5,000 reviews, summarise 300 documents, run an evaluation set. Done one after another, 5,000 calls of 1 second each take well over an hour. Done all at once, they trip rate limits (Topic 6.7) and may crash your own process. You want a middle: a fixed number of calls in flight at the same time.
- 2
The concurrency limit (a semaphore). A semaphore is a counter of free places. A call must take a place to start and gives it back when it finishes. With a limit of 3, the fourth call waits until one of the first three ends. The example runs eight calls of different lengths on a simulated clock: limit 1 takes 2,700 ms, limit 3 about a third of that, and limit 8 takes as long as the slowest call (600 ms). The simulation uses a scheduler loop, so even the Java version is single-threaded.
- 3
Choose the limit from the quota. The limit should keep you under your provider's requests-per-minute and tokens-per-minute limits (Topic 6.7), leave room for other users of the same key, and match what your own machine can hold open. If calls take 2 seconds and you may send 300 per minute, about 10 in flight uses the quota fully. Combine the limit with a token bucket and retries for 429s.
- 4
Keep results in order. Calls finish in a different order than they started: the example's finish order is not 0, 1, 2…. If you append results as they finish, the output list no longer matches the input list. Store each result by its position (or by an id), then read in the original order. The same rule saves you in batch APIs, where each request carries a
custom_id. - 5
Failures inside a pool. One failed call out of 5,000 should not lose the other 4,999. Catch errors per call, keep the call's id, retry it with backoff (Topic 6.8) and report what still failed at the end. Decide up front whether one failure stops the whole job or not.
- 6
Batch APIs for offline work. If nobody is waiting (nightly summaries, labelling a dataset, evaluations), many providers offer a batch API: you upload a file of requests, results are ready later (often within hours), and the price is often lower than for live calls. The trade is waiting. Check your provider's current price, delay and limits, since all of them differ and change. The second example shows the pattern: send requests with ids, match results by id, find the missing ones and send those again.
Explain it without notes
What is a concurrency limit and how do you choose its value?
Why can results come back in a different order than the requests, and how do you keep them aligned?
When would you use a batch API and what are its trade-offs?
How should one failed call inside a pool of thousands be handled?
Practice
Write makespan(durations, limit) returning the finish time when the calls start in list order and at most limit run at once (the clock jumps to the next finish). Print it for [300, 500, 200, 400] with limits 1 and 2.
Write put_back_in_order(requests, results) where results is a list of {"id": ..., "text": ...} in any order and requests is a list of ids. Return the texts in the order of requests, with "MISSING" for an id with no result.
Trade-offs
- ↔
A higher limit finishes sooner and risks 429s and memory; a lower one is gentle and slow.
- ↔
Batch APIs are cheaper and slower, and live calls are quick and pricier.
- ↔
Resumable jobs need storage and ids; they save money when something goes wrong.
Done when you can
Done when you can write a concurrency-limited scheduler and explain the limit.
Done when you store results by position or id.
Done when you can explain when a batch API is the right tool.
Done when failures in a pool are caught, retried and reported without losing other results.