PHASE 12Advanced Python 3.11+ ~28 min· topic 7 of 9

Topic 12.7

Running Things Together: gather, TaskGroup, Timeouts

In one line

asyncio.gather and TaskGroup (Python 3.11) run many coroutines together, asyncio.timeout (Python 3.11) stops slow work, cancellation lets you stop tasks safely, a Semaphore limits how many run at once, and to_thread hands blocking code to a thread. These are the building blocks of every async LLM client.

Think of it like this

A teacher collecting homework from a class. She can ask every student at the same time (gather), but if one student finds a mistake in the assignment itself she wants everyone to stop and the problem reported (TaskGroup). She may say "anyone not done in five minutes, hand in what you have" (timeout), or "only two students at the photocopier at once" (semaphore).

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

gather: Run several awaitables together and collect their results in argument order.
TaskGroup: A block that owns a group of tasks, waits for them all, and cancels the rest if one fails.
ExceptionGroup: One exception that holds several others. Caught with except*.
Structured concurrency: A rule that tasks may not outlive the block that started them.
Timeout: A limit on how long a block may take before it is cancelled.
Cancellation: Stopping a task by raising CancelledError inside it.
Semaphore: A counter that lets only n tasks into a section at once.
Rate limit: A rule from an API about how many requests you may send in some period or at once.
to_thread: Runs a blocking function in a worker thread and lets you await it.

Step by step

01gather: results in argument order

Three fake model calls take 0.06, 0.02 and 0.04 seconds. gather waits for all and returns ['A: ok', 'B: ok', 'C: ok']: the order of the arguments, even though B finished first. With return_exceptions=True, a failure comes back as an object in the list (here RuntimeError('B failed')) and the good results are kept.

02TaskGroup: fail together, nothing lost

In the group, one task is slow (0.2 s) and one fails after 0.02 s. When bad fails, the group cancels slow (so slow.cancelled() is True), waits until it stops, and raises an ExceptionGroup. except* RuntimeError catches the errors of that type; errors.exceptions lists them.

Compare with gather: with gather the slow task would keep running in the background after the error. A TaskGroup never leaves one behind.

TaskGroup: fail together, nothing lostdiagram

Rendering diagram…

03Timeouts and cancellation

asyncio.timeout(0.05) cancels slow_model after 0.05 seconds by raising CancelledError inside it. The model's except and finally blocks run (so cleanup happens), the error is re-raised, and the timeout turns it into TimeoutError outside. A manual task.cancel() works the same way; awaiting the cancelled task raises CancelledError.

Note two things in the code: the finally block always runs, and the except asyncio.CancelledError re-raises. If it did not re-raise, the timeout would never fire.

04Semaphore: five calls, two at a time

The example keeps its counters in nonlocal variables, which let an inner function change a name of the outer function (Topic 5.5). All five coroutines are created by gather, but each must enter async with limit: first. Calls 0 and 1 enter; 2, 3 and 4 wait. When a call finishes, the next one enters. peak records the most calls inside at once, and it never goes above 2. This is the standard shape of a rate-limited batch of LLM calls.

Semaphore: five calls, two at a timediagram

Rendering diagram…

05to_thread for blocking SDKs

When a library only offers a blocking call, await asyncio.to_thread(fn, arg) runs it in a worker thread. The three calls overlap in three threads, and a heartbeat task keeps ticking, proving the loop is free. This needs real threads, so the Run button here cannot show it: the browser's event loop runs the function on the same thread, which would block. The output below is from a normal Python 3.12.

06Putting it together

A typical LLM batch is: a semaphore for the limit, a timeout around each call, a TaskGroup or gather(..., return_exceptions=True) to collect everything, and to_thread for any blocking SDK. Topic 12.9 builds exactly this, with retries.

Try it yourself

1
Raise the limit
In the semaphore example change Semaphore(2) to Semaphore(5) and then to Semaphore(1). Predict the order of the "starts" lines and the value of peak each time. With 1, the calls run strictly one by one.
2
Swallow the cancellation
In the timeout example delete the raise after the cleanup message (and the finally, if you like) so the except block swallows CancelledError. What does the caller print now, and what does task.cancelled() say? A task that hides its cancellation looks like it finished normally, which breaks timeouts.
3
Catch it your way
In the first example change except* RuntimeError to except* ValueError. What happens to the error? An ExceptionGroup that is not handled by any except* clause keeps travelling up (and the run ends with a traceback).

Code & diagrams

gather, return_exceptions and TaskGroup Python 3.11+ New tab

TaskGroup and except* need Python 3.11 or newer. The browser runs 3.13.

Expected output

gather keeps argument order: ['A: ok', 'B: ok', 'C: ok']
with return_exceptions: ["'A: ok'", "RuntimeError('B failed')"]
group raised: ['bad failed']
slow task cancelled: True

Timeout and cancellation Python 3.11+ New tab

asyncio.timeout is Python 3.11. The model prints while it is being cancelled, so you can see the cleanup run.

Expected output

model: thinking
model: cancelled, cleaning up
model: finally always runs
caller: timed out
model: thinking
model: cancelled, cleaning up
model: finally always runs
caller: task.cancelled() is True

A Semaphore keeps the number of calls in flight under a limit New tab

Five calls, at most two inside the semaphore at once. We print the peak, not any timing.

Expected output

call 0 starts
call 1 starts
call 2 starts
call 3 starts
call 4 starts
results: [0, 1, 4, 9, 16]
most calls at once: 2

to_thread keeps the loop free (run on a normal Python)python

Not runnable in the browser: it needs real threads. In the browser, the same call would run on the main thread and block. Verified with Python 3.12 on a desktop. The heartbeat flag is True because the loop kept running while three 0.1 s blocking calls ran in threads.

import asyncio
import time


def blocking_sdk_call(x):
    time.sleep(0.1)                       # an old library that cannot await
    return f"sdk result {x}"


async def main():
    beats = 0

    async def heartbeat():
        nonlocal beats
        while True:
            await asyncio.sleep(0.01)
            beats += 1

    pulse = asyncio.create_task(heartbeat())
    results = await asyncio.gather(*(asyncio.to_thread(blocking_sdk_call, i) for i in range(3)))
    pulse.cancel()
    print(results)
    print("event loop stayed responsive:", beats >= 3)


asyncio.run(main())

Output

['sdk result 0', 'sdk result 1', 'sdk result 2']
event loop stayed responsive: True

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Use asyncio.timeout with a plain with

Write with asyncio.timeout(1): instead of async with asyncio.timeout(1):.

main.pywhole filepython

import asyncio


async def main():
    with asyncio.timeout(1):
        await asyncio.sleep(0.01)


asyncio.run(main())

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 9, in <module>

asyncio.run(main())

^^^^^^^^^^^^

File "main.py", line 5, in main

with asyncio.timeout(1):

~~~~~~~~~~~~~~~^^^

TypeError: 'Timeout' object does not support the context manager protocol

Myth vs fact

Myth

gather cancels the other tasks when one fails.

Fact

It raises the first error to you, but the other awaitables keep running. Use a TaskGroup when you want the rest cancelled.

Myth

A semaphore limits requests per second.

Fact

It limits how many are in progress at once. A per-second limit also needs spacing between starts.

Myth

A timeout stops any code after the time is up.

Fact

It cancels the task at its next await. Code that never awaits (a tight loop) cannot be interrupted by a timeout.

Myth

asyncio.to_thread makes slow Python code fast.

Fact

It moves the code off the event loop thread so others keep running. CPU-bound Python still shares one core because of the GIL.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
ExceptionGroup flattens: if a TaskGroup has several failing tasks you get all of them in errors.exceptions, and except* can handle different types in different clauses. KeyboardInterrupt and SystemExit are never grouped. TaskGroup and asyncio.timeout arrived in Python 3.11; code for older versions uses gather and asyncio.wait_for.
▸
With asyncio.timeout, the deadline can be moved while the block runs (cm.reschedule(...)), and asyncio.timeout_at takes an absolute loop time. Nested timeouts work: the earliest deadline wins, and each timeout converts only its own cancellation into TimeoutError.
▸
A rate limit is usually per second (or per minute, or per token count), so production LLM clients combine a semaphore with a token bucket, a retry with exponential backoff (each wait twice as long as the last) and jitter on 429 ("too many requests") and 5xx (server error) responses, and a timeout per call. Respect the Retry-After header (a line in the reply that says how many seconds to wait) when the API sends one.
▸
Cleanup during cancellation can itself await, but a second cancellation may arrive while cleaning up. For cleanup that must finish, use asyncio.shield(...) sparingly, and never leave a shielded task without a reference.
▸
asyncio.to_thread uses the loop's default thread pool, which has at most min(32, os.cpu_count() + 4) threads, so at most that many blocking calls run at once and the rest queue. It also copies the current contextvars context into the thread. For a different size, create your own ThreadPoolExecutor and use loop.run_in_executor. A cancelled to_thread await stops waiting, but the thread itself keeps running until the function returns.
▸
Cancelling a task that is awaiting gather cancels all the children it gathered. A TaskGroup whose parent is cancelled cancels its children and waits for them. Code that catches CancelledError to do cleanup should use finally instead where possible. asyncio.Lock, Semaphore and Queue are not thread-safe: use them from the event loop's thread only.

Remember this

1
**asyncio.gather(*coros) runs all the coroutines together and returns a list of results in the same order as the arguments**, not the order they finished. If one raises, gather raises that error to you at once, but the other tasks keep running unless you cancel them. With return_exceptions=True it never raises: failed jobs appear in the result list as exception objects.
2
**asyncio.TaskGroup** (Python 3.11) is the modern, safer way. Inside async with asyncio.TaskGroup() as group: you call group.create_task(coro). (async with is a with block that is allowed to wait, and Topic 12.8 explains it. For now read it as the with from Topic 7.4.) When the block ends, it waits for all tasks. If one task fails, the group cancels all the others, waits for them to stop, and raises an **ExceptionGroup** with the errors. Catch it with except* (also 3.11; recall Topic 7.3: an ExceptionGroup is one error that holds several errors). This is called structured concurrency: tasks cannot outlive the block that started them, so nothing is silently lost.
3
Timeouts: async with asyncio.timeout(seconds): (Python 3.11) cancels the block if it takes too long and raises TimeoutError outside it. On older versions the equivalent is asyncio.wait_for(coro, seconds). Always set a timeout on network calls: a missing reply otherwise hangs a task forever, and with it a whole batch.
4
Cancellation works by raising asyncio.CancelledError inside the task at its current await. A well-behaved task cleans up (in finally or async with) and lets the error continue. Do not swallow CancelledError with a bare except: or except Exception that hides it (in modern Python it is a BaseException, so except Exception does not catch it, but except BaseException does). A task that ignores cancellation can never be stopped, and timeouts depend on it.
5
**asyncio.Semaphore(n)** lets at most n coroutines inside async with sem: at once; the rest wait their turn. This is how you respect an API's rate limit or connection limit while still sending hundreds of requests: all are scheduled, but only n are in flight. A semaphore limits concurrency, not rate per second; a per-second limit needs a token bucket (a counter that refills at a steady rate, where each request uses one) or a sleep between starts.
6
**asyncio.to_thread(fn, *args)** runs a normal blocking function in a worker thread and lets you await its result, so the event loop stays free. Use it for an SDK that has no async version, a blocking file read, or a CPU-light but slow call. It does not speed up CPU-bound Python (the GIL, Topic 12.2). To use it from the browser see the note in the walkthrough.

Explain it without notes

What is the difference between asyncio.gather and asyncio.TaskGroup when one task fails?

How does cancellation work in asyncio, and why must you re-raise CancelledError?

How do you limit a batch of 500 LLM calls to 10 at a time?

When do you use asyncio.to_thread, and what does it not do?

A coroutine is awaiting asyncio.gather(a(), b(), c()) and the coroutine itself gets cancelled. What happens to a, b and c? And what if b raises instead?

Practice

Use gather to run double(1), double(2), double(3) (each sleeps 0.01 and returns n * 2) and print the list.

Write first_with_timeout(seconds), which runs asyncio.sleep(0.2) inside asyncio.timeout(seconds) and returns "done" or "timed out". Test it with 0.05 and 0.5.

Make an asyncio.Lock() and a counter that starts at 0. Write a coroutine add_one that, inside async with lock:, reads the counter, runs await asyncio.sleep(0.01), then stores the read value plus 1. Run five of them with gather and print the final counter (it should be 5).

Trade-offs

↔
gather vs TaskGroup: gather is short and works on every version; TaskGroup (3.11+) cancels siblings and never loses an error, at the cost of ExceptionGroup handling.
↔
Large vs small semaphore: a larger limit finishes sooner, but triggers rate-limit errors (HTTP 429) and uses more memory and connections.
↔
Timeouts: short timeouts fail fast but may cut off healthy slow calls, such as long model generations; choose them from the call's real behaviour.

Done when you can

Done when you can run several coroutines with gather and say in what order the results come back.
Done when you can use a TaskGroup and catch its errors with except*.
Done when you can put a timeout around a call and explain how cancellation reaches the task.
Done when you can limit concurrency with a Semaphore and explain what it does not limit.
Done when you can explain when to_thread helps and when it does not.

Command Palette

Words you'll meet

Step by step

01gather: results in argument order

02TaskGroup: fail together, nothing lost

03Timeouts and cancellation

04Semaphore: five calls, two at a time

05to_thread for blocking SDKs

06Putting it together

Try it yourself

Raise the limit

Swallow the cancellation

Catch it your way

Code & diagrams

Break it on purpose

Use asyncio.timeout with a plain with

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can