A batch: send with ids, match by id
The 50 percent price and the price per token are example numbers. A batch API has its own price, delay and limits, so read your provider's terms.
Running many model calls at once finishes a big job much sooner, but too many at once hits rate limits. A concurrency limit keeps a fixed number in flight, results are stored by position so order survives, and batch APIs handle offline work more cheaply.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write makespan(durations, limit) returning the finish time when the calls start in list order and at most limit run at once (the clock jumps to the next finish). Print it for [300, 500, 200, 400] with limits 1 and 2.
Write put_back_in_order(requests, results) where results is a list of {"id": ..., "text": ...} in any order and requests is a list of ids. Return the texts in the order of requests, with "MISSING" for an id with no result.
Explain it without notes
What is a concurrency limit and how do you choose its value?
Why can results come back in a different order than the requests, and how do you keep them aligned?
When would you use a batch API and what are its trade-offs?
How should one failed call inside a pool of thousands be handled?
Expected output
came back in this order: doc-4, doc-2, doc-1, doc-0
doc-0: ALPHA
doc-1: BRAVO
doc-2: CHARLIE
doc-3: MISSING
doc-4: ECHO
send again in the next batch: doc-3
cost if sent one by one: 20000 micro-dollars, as a batch: 10000 micro-dollars