Command Palette

Search for a command to run...

Hectal
Phase 12Advanced13 of 18 in Python

Concurrency and Async

Threads, processes and asyncio: what the GIL really does, how to run hundreds of LLM calls at once, and how to choose the right tool without guessing.

A program that does one thing at a time wastes most of its life waiting: for a web server, a database, a disk, an AI model. This phase teaches the three ways Python can do more than one thing at once (threads, processes and asyncio), what each one costs, and how to choose between them.

asyncio is the heart of the phase, and it runs live on this page. You will write coroutines, run them together with gather and TaskGroup, limit them with a Semaphore, stop them with timeouts, and stream fake model tokens with an async generator. This is exactly the shape of code that calls LLM APIs in production.

Threads and processes cannot start inside a web page, so those topics show real programs that were run on a normal Python, plus live examples of everything around them that does run here (locks, queues, futures, pickling). The phase ends with a decision guide and a debugging checklist.

0/9 · 0%
9 topics ~4 h 35 code blocks & diagrams
Start with the first topic
1
12.1

Concurrency vs Parallelism: Threads, Processes, Async

Concurrency means several tasks are in progress at once, even if only one runs at any instant; parallelism means several really run at the same instant. Python gives you three tools (threads, processes and asyncio), and the right one depends on whether your program mostly waits (I/O-bound) or mostly calculates (CPU-bound).

28 min 1 diagram 3 code practice

2
12.2

The GIL and Free-Threaded Python

The Global Interpreter Lock (GIL) lets only one thread run Python bytecode at a time in the standard CPython build, so threads do not speed up pure-Python calculation but still help with waiting. A free-threaded build without the GIL exists (experimental in 3.13, officially supported in 3.14).

26 min 4 code practice

3
12.3

Threads with threading

A thread is a flow of code that runs inside your process and shares its memory. You start threads with threading.Thread, wait for them with join, and protect shared data with a Lock; queues and thread-local data are the safer ways to share and separate state.

31 min 5 code practice

4
12.4

Thread and Process Pools: concurrent.futures

concurrent.futures gives you a pool of workers and a Future for every job you submit. ThreadPoolExecutor suits blocking I/O, ProcessPoolExecutor suits CPU-bound work, and both share the same simple API: submit, map and as_completed.

27 min 4 code practice

5
12.5

multiprocessing

multiprocessing runs your Python code in separate processes, each with its own interpreter, its own GIL and its own memory, so CPU-bound work can use every core. The price is that arguments and results are copied by pickling, and your script must be safe to import again.

29 min 4 code practice

6
12.6

asyncio: async and await

asyncio runs many tasks on one thread. A function defined with async def is a coroutine that can pause at await, and the event loop (started by asyncio.run) runs whichever task is ready while the others wait.

27 min 3 code practice

7
12.7

Running Things Together: gather, TaskGroup, Timeouts

asyncio.gather and TaskGroup (Python 3.11) run many coroutines together, asyncio.timeout (Python 3.11) stops slow work, cancellation lets you stop tasks safely, a Semaphore limits how many run at once, and to_thread hands blocking code to a thread. These are the building blocks of every async LLM client.

28 min 4 code practice

8
12.8

Async Iterators, Generators and Context Managers

async for loops over values that arrive over time, an async generator (async def with yield) produces them, async with opens and closes resources that need to await, and asyncio.Queue passes work between tasks. Streaming model tokens is the classic use.

29 min 3 code practice

9
12.9

Choosing and Debugging Concurrency

Pick the model from the bottleneck: many API calls mean asyncio, a blocking library means threads, CPU-bound Python means processes, and array maths means NumPy. This topic gives the decision guide, a full async pipeline with a limit, timeout and retries, and the bugs that most often go wrong.

28 min 1 diagram 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Fetch many pages at once with gather

fake_fetch(name) pretends to download a page and returns "page:<name>". Write async def fetch_all(names) that fetches all the names at the same time with asyncio.gather and returns the list of pages in the same order as names.

Intermediate
2

Collect results and errors without crashing

risky(name) returns name.upper(), but raises ValueError for names that start with "bad". Write async def collect_all(names) that runs risky for every name together and returns one string per name, in order: "ok:<RESULT>" for a success or "error:<message>" for a failure.

Intermediate
3

Give up on slow calls with a timeout

Write async def answer_or_default(make_coro, seconds, default). It awaits make_coro(), but if that takes longer than seconds it stops waiting and returns default instead.

Advanced
4

Run jobs with a concurrency limit

Write async def run_limited(factories, limit). factories is a list of functions that take no arguments and each return a coroutine (for example lambda: job(3); calling it gives the coroutine to await). Run them all together but with **at most limit running at once** (use asyncio.Semaphore), and return the results in the order of factories.

Advanced
5

Transform a token stream with an async generator

words(text) is an async generator that yields the words of a text, one at a time, like a model streaming tokens. Write two things: shout(stream), an async generator that yields each item of another async stream in upper case, and async def collect(stream) that returns all items of an async stream as a list.

Advanced
6

Process a batch with a queue and workers

square(n) returns n * n after a short wait. Write async def process_all(items, workers) that puts all items in an asyncio.Queue, starts workers worker tasks that take items from the queue and call square, waits until every item is handled (queue.join()), stops the workers, and returns the squares sorted from smallest to largest.

Advanced