PHASE 12Advanced ~27 min· topic 4 of 9

Topic 12.4

Thread and Process Pools: concurrent.futures

In one line

concurrent.futures gives you a pool of workers and a Future for every job you submit. ThreadPoolExecutor suits blocking I/O, ProcessPoolExecutor suits CPU-bound work, and both share the same simple API: submit, map and as_completed.

Think of it like this

A restaurant kitchen with four cooks and a rail of order tickets. You do not hire a new cook for every order; the cooks take tickets one after another. When you hand in an order you get a numbered receipt. The food is not ready yet, but the receipt lets you ask later: "is it done? what was the result? did something go wrong?" The cooks are the pool; the receipt is a Future.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Executor / pool: An object that owns a set of worker threads or processes and runs submitted jobs on them.
Future: A placeholder for a result that may not be ready yet. It can later hold a value or an exception.
submit: Queue one job and get its Future back immediately.
map: Run a function over many items; results come back in input order.
as_completed: Iterate over futures in the order they finish.
Callback: A function that is called automatically when something finishes.
Pickling: Turning an object into bytes so it can be sent to another process.
max_workers: The most jobs the pool runs at the same time.

Step by step

01Futures are just objects

A Future does not need a pool to exist. You can create one, give it a result with set_result or an error with set_exception, and read it back. This is what an executor does for you behind the scenes, and it runs live here because no thread is needed.

Notice the callback: add_done_callback runs as soon as the future gets its result. And cancel() works only on a future that has not started.

02Executors on this page

ThreadPoolExecutor and ProcessPoolExecutor both need real threads or processes, so the Run button here cannot use them (the browser sandbox does not allow it, Topic 12.3). The examples below were run on Python 3.12 on a desktop and the output is pasted. In the browser, the same API is available as asyncio (Topics 12.6 to 12.8), which is the better tool for many network calls anyway.

03submit and as_completed

Here four jobs are submitted. Job n sleeps longer for smaller n, so they finish in the order 4, 3, 2, 1. as_completed hands us the futures in that order. The dictionary futures maps each future back to its input so we know which job each one was. (It is built with a dict comprehension, Topic 4.7.) Job 3 raises, and we catch the error at result().

main.pywhole filepython

with ThreadPoolExecutor(max_workers=4) as pool:
    futures = {pool.submit(fetch, n): n for n in [1, 2, 3, 4]}
    for fut in as_completed(futures):
        n = futures[fut]
        try:
            print("item", n, "->", fut.result())
        except ValueError as error:
            print("item", n, "failed:", error)

04map keeps the input order

pool.map(f, [1, 2, 3]) returns results in input order even if the third job finished first. The call returns an iterator right away; asking for the next value waits for that particular job. Use map when you only need the values, submit plus as_completed when you need to react to each one as it finishes or to know which input failed.

05Processes: the same API, different cost

ProcessPoolExecutor has the same methods. The differences are practical: workers start slowly (each is a new interpreter), inputs and outputs are pickled and copied, and on platforms that use spawn your script is imported again in each worker, so the pool must be created under if __name__ == "__main__":. Topic 12.5 explains why.

Processes: the same API, different costdiagram

Rendering diagram…

06Pools and asyncio together

Async code can hand a blocking function to a thread pool without blocking the event loop: await loop.run_in_executor(pool, fn, arg), or the shorter await asyncio.to_thread(fn, arg) (Topic 12.7). A process pool works with run_in_executor too, for heavy maths inside an async server.

Try it yourself

1
Cancel and wait
In the first example call f.cancel() before set_result. What does cancel() return, and what happens to the callback? A cancelled future never gets a result, so a call to set_result on it raises an error.
2
Predict the as_completed order
In the thread-pool example change the sleep to 0.1 * n and predict the new order before running it on your computer. Then set max_workers=1 and explain why the order changes again: with one worker, jobs run one by one in submission order.
3
Size a pool
Suppose an API allows 20 requests at a time and each takes about a second. You have 200 items. How long do you expect with 20 workers (about 10 seconds), and with 100 workers (the server rejects some)? Pool size should follow the limit of what you are waiting for.

Code & diagrams

Futures made by hand New tab

No threads needed: a Future is just a box with a state. An executor is a machine that fills these boxes.

Expected output

new future -> done: False | cancelled: False
callback sees: 42
after set_result -> done: True | result: 42
exception stored: ValueError('model said no')
result() raised: model said no
cancel a pending future: True | cancelled: True
result() raised CancelledError
cancel a finished future: False

wait, as_completed and timeouts on futures New tab

Two futures are finished and one is pending. We sort the names because the order a set gives them is not guaranteed.

Expected output

done: 2 | not done: 1
as_completed yielded: ['a', 'c']
b is not ready: TimeoutError

ThreadPoolExecutor: map, submit, as_completed (run on a normal Python)python

Not runnable in the browser. Verified with Python 3.12 on a desktop. The jobs finish in order 4, 3, 2, 1 because bigger n sleeps less, with 0.1 s gaps.

import time
from concurrent.futures import ThreadPoolExecutor, as_completed


def fetch(n):
    time.sleep(0.1 * (4 - n))          # a bigger n finishes sooner
    if n == 3:
        raise ValueError(f"bad input {n}")
    return n * n


with ThreadPoolExecutor(max_workers=4) as pool:
    print("map:", list(pool.map(lambda n: n * 10, [1, 2, 3])))
    futures = {pool.submit(fetch, n): n for n in [1, 2, 3, 4]}
    for fut in as_completed(futures):
        n = futures[fut]
        try:
            print("item", n, "->", fut.result())
        except ValueError as error:
            print("item", n, "failed:", error)

Output

map: [10, 20, 30]
item 4 -> 16
item 3 failed: bad input 3
item 2 -> 4
item 1 -> 1

ProcessPoolExecutor and the lambda problem (run on a normal Python)python

Not runnable in the browser. Verified with Python 3.12 on Windows. The guard under __main__ is required. A lambda cannot be pickled, so the job fails with a PicklingError.

from concurrent.futures import ProcessPoolExecutor


def square(n):
    return n * n


if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(list(pool.map(square, range(5))))
        try:
            pool.submit(lambda n: n, 1).result()
        except Exception as error:
            print(type(error).__name__)

Output

[0, 1, 4, 9, 16]
PicklingError

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Ask a pending future for its result with no time to wait

Create a Future that nobody ever fills in and call result(timeout=0).

main.pywhole filepython

from concurrent.futures import Future

pending = Future()
pending.result(timeout=0)

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 4, in <module>

pending.result(timeout=0)

~~~~~~~~~~~~~~^^^^^^^^^^^

File "/lib/python313.zip/concurrent/futures/_base.py", line 458, in result

raise TimeoutError()

TimeoutError

Myth vs fact

Myth

If a job in the pool fails, my program crashes.

Fact

The exception is stored in the Future. It only appears when you call result() or exception(). If you never look, you never know.

Myth

pool.map returns results as they finish.

Fact

It returns them in input order. Use submit with as_completed for completion order.

Myth

A process pool is just a faster thread pool.

Fact

It escapes the GIL for CPU-bound work, but it costs startup time and copies every argument and result, so for small jobs it is often slower.

Myth

More workers is always better.

Fact

Past the limit of the CPU, the network, the memory or the server's rate limit, extra workers only add contention and failures.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Executor.shutdown(wait=True, cancel_futures=True) (the cancel_futures argument is available since Python 3.9) cancels jobs that have not started; running jobs still finish because Python cannot safely kill a thread. Leaving a with block is shutdown(wait=True), so a hung job hangs your exit.
▸
ProcessPoolExecutor raises BrokenProcessPool if a worker dies abruptly (out of memory, a native crash). Treat it as fatal for that pool and create a new one. Large arguments and results are pickled through a pipe, so passing a big NumPy array to every job can cost more than the work; send an index or a path, or use shared memory (Topic 12.5).
▸
chunksize on ProcessPoolExecutor.map batches many small items into one message and removes most per-item overhead. For thousands of tiny tasks, chunk them.
▸
Exceptions cross process boundaries by pickling too, so a custom exception with a required constructor argument may fail to unpickle and surface as a confusing error. Keep exceptions used in workers simple.
▸
A done-callback runs in whichever thread completes the future (a worker thread, or the calling thread if the future is already done), so keep callbacks short and thread-safe. as_completed(futures, timeout=...) and wait(..., timeout=...) raise or return early without cancelling anything: the jobs keep running. Pool threads are joined at interpreter exit, so a hung job also hangs shutdown.
▸
ProcessPoolExecutor(max_tasks_per_child=n) (Python 3.11) replaces a worker after n jobs, which contains slow memory leaks in native libraries. mp_context=multiprocessing.get_context("spawn") pins the start method, and initializer= runs setup code once per worker, the right place to load a model or open a connection instead of doing it per job.

Remember this

1
A pool (an *executor*) keeps a fixed number of workers alive and feeds them jobs. This avoids the cost of starting a thread or process per job, and it puts a ceiling on how many run at once. Create one with with ThreadPoolExecutor(max_workers=8) as pool:; leaving the with block waits for all submitted jobs to finish and shuts the workers down.
2
pool.submit(fn, *args) returns a Future straight away. A Future is a placeholder for a result that may not exist yet. future.result() waits for the job and returns its value, or re-raises the exception the job raised. Other methods: done(), exception(), cancel() (works only if the job has not started), add_done_callback(fn).
3
pool.map(fn, items) is the shortcut: it runs fn on every item and returns the results in the order of the input, not the order they finish. If one call raised, the exception shows up when you reach that result while iterating. concurrent.futures.as_completed(futures) is the opposite: it gives you each future as soon as it finishes, so you can show progress or handle failures early. wait(futures, return_when=...) blocks until some or all are done.
4
Exceptions are stored, not thrown at you. A job that fails does not crash your main program. The error sits inside its Future until you call result() or exception(). If you submit jobs and never look at the futures, failures disappear silently, so always collect and check results.
5
Which pool? ThreadPoolExecutor for blocking I/O: network calls, files, database drivers that have no async version. ProcessPoolExecutor for CPU-bound pure-Python work, because each worker is a separate process with its own GIL (Topic 12.5). With processes, the function and arguments are pickled (converted to bytes) and sent over, so they must be picklable, and a lambda (a small unnamed function, Topic 5.6) is not (Topic 12.5).
6
Sizing a pool. For I/O-bound threads, the right number is about how many requests the other side will tolerate at once (often 8 to 64), not the number of cores. For processes it is about os.cpu_count(), since more than that gives no extra speed. The default for threads is min(32, os.cpu_count() + 4). Bigger pools are limited by rate limits, memory and the server on the other end.

Explain it without notes

What is a Future, and how does result() treat exceptions?

How do map and as_completed differ in the order of results?

When would you choose ThreadPoolExecutor over ProcessPoolExecutor, and the other way round?

How do you decide the size of a pool?

Inside a with ThreadPoolExecutor() block your code raises an exception after submitting 100 jobs. What happens to the jobs, and how would you stop them sooner?

Practice

Create a Future, attach a callback that prints got 7, then set its result to 7.

Write first_error(futures) that returns the first exception stored in a list of finished futures, or None.

Write all_results(futures) that returns a list of results, with "failed" in place of any future that holds an exception.

Trade-offs

↔
submit + as_completed: full control and early reaction, but you manage the future-to-input mapping yourself.
↔
map: the shortest code, but results come in input order and one failure interrupts the iteration.
↔
Thread vs process pool: the same code shape with very different costs; choose by whether the job waits or calculates.

Done when you can

Done when you can explain a Future with the numbered-receipt analogy.
Done when you can use submit, result, map and as_completed and say how their result order differs.
Done when you can explain why an exception inside a job only appears at result().
Done when you can choose between a thread pool and a process pool for a given job.
Done when you can give a reason for the size you choose for a pool.

Command Palette

Words you'll meet

Step by step

01Futures are just objects

02Executors on this page

03submit and as_completed

04map keeps the input order

05Processes: the same API, different cost

06Pools and asyncio together

Try it yourself

Cancel and wait

Predict the as_completed order

Size a pool

Code & diagrams

Break it on purpose

Ask a pending future for its result with no time to wait

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can