A concurrency limit, simulated
A scheduler loop on a simulated clock. Real code would use asyncio or a thread pool; the logic of the limit is the same.
Running many model calls at once finishes a big job much sooner, but too many at once hits rate limits. A concurrency limit keeps a fixed number in flight, results are stored by position so order survives, and batch APIs handle offline work more cheaply.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write makespan(durations, limit) returning the finish time when the calls start in list order and at most limit run at once (the clock jumps to the next finish). Print it for [300, 500, 200, 400] with limits 1 and 2.
Write put_back_in_order(requests, results) where results is a list of {"id": ..., "text": ...} in any order and requests is a list of ids. Return the texts in the order of requests, with "MISSING" for an id with no result.
Explain it without notes
What is a concurrency limit and how do you choose its value?
Why can results come back in a different order than the requests, and how do you keep them aligned?
When would you use a batch API and what are its trade-offs?
How should one failed call inside a pool of thousands be handled?
Expected output
limit 1: all done at 2700 ms, at most 1 at once
finished in this order: 0 1 2 3 4 5 6 7
limit 3: all done at 1000 ms, at most 3 at once
finished in this order: 2 0 4 1 3 6 7 5
limit 8: all done at 600 ms, at most 8 at once
finished in this order: 4 2 6 0 7 3 1 5
results in request order: answer-0, answer-1, answer-2, answer-3, answer-4, answer-5, answer-6, answer-7