PHASE 12Advanced ~29 min· topic 5 of 9

Topic 12.5

multiprocessing

In one line

multiprocessing runs your Python code in separate processes, each with its own interpreter, its own GIL and its own memory, so CPU-bound work can use every core. The price is that arguments and results are copied by pickling, and your script must be safe to import again.

Think of it like this

Instead of one cook sharing a kitchen, you open several separate kitchens, each with its own stove, pans and cookbook. They really cook at the same time and cannot get in each other's way. But they also cannot reach into each other's fridge: to share anything you must carry it over in a box and the other kitchen gets a copy. Processes are separate kitchens; threads were cooks sharing one.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Process: A running program with its own memory and its own interpreter.
Pickle: Python's way of turning an object into bytes and back, so it can be sent to another process or saved.
Picklable: Able to be pickled. Lambdas, local functions, locks and open files are not.
Pool: A fixed group of worker processes that share a list of jobs.
Serialise: Turn an object into bytes (or text) so it can be sent or saved. Pickling is Python's way of doing it.
spawn: Start a new, empty interpreter and import your script. Default on Windows and macOS.
fork: Start a new process by copying the parent's memory. Fast, but unsafe with threads. Unix only (Linux and macOS, not Windows).
forkserver: A start method where a clean helper process forks the workers. Default on Linux from Python 3.14.
Shared memory: A block of memory that several processes can read and write.
Main guard: The line if __name__ == "__main__": that keeps code from running when the file is imported.

Step by step

01Separate kitchens, separate memory

The Run button on this page cannot start a process (the browser has no way to create one), and Process.start() fails with an OSError. Everything about processes below that needs a real process was run on a desktop Python, and the output is pasted. The parts about pickling and copying run live.

Separate kitchens, separate memorydiagram

Rendering diagram…

02What a worker actually receives

A worker never sees the parent's variable, only bytes that describe it. It builds its own copy, changes the copy, and sends bytes back. The next example does that by hand in one process: the parent's list stays untouched, and the reply is a different list.

main.pywhole filepython

def worker(request_bytes):
    data = pickle.loads(request_bytes)   # the worker's own copy
    data.append("worker was here")
    return pickle.dumps(data)

03What can be pickled

The pickle of a top-level function is just its name and module (tools.square). The worker imports that module and looks the name up, so the function must exist at the top level of an importable module. A lambda has no importable name, and a function defined inside another function has none either, so both fail. A lock fails because an operating-system object cannot be turned into bytes.

The error types differ: a lambda raises PicklingError, a local function raises AttributeError (in the Python versions tried here), and a lock raises TypeError. Do not rely on the exact type; rely on the rule.

04Pool.map and the main guard

The program below creates a pool of two workers and maps square over six numbers. It also tries to change a global counter in the workers. The parent's counter stays 0, which proves that memory is not shared. The pool is created under the __main__ guard, because spawn re-imports the file in every worker, and everything outside the guard runs again there.

main.pywhole filepython

if __name__ == "__main__":
    ctx = mp.get_context("spawn")
    with ctx.Pool(processes=2) as pool:
        print("map:", pool.map(square, range(6)))
        pool.map(bump, [1, 2, 3])
    print("parent's counter:", counter)

05spawn, fork and forkserver

With fork the child starts as a copy of the parent's memory (instant, and it can use any function the parent had). With spawn the child starts empty and imports your script (slower, but clean and the same on every operating system). With forkserver a small clean server process is started once and forks the workers on request.

Defaults: Windows and macOS use spawn. Linux used fork up to Python 3.13 and uses forkserver from Python 3.14. Code that worked only under fork (for example, passing a local function) breaks under spawn, so write for spawn.

06Sharing on purpose

Value and Array keep a small shared number or array; shared_memory.SharedMemory shares a raw block by name. Two processes adding to one shared counter must still lock, as threads do. The second desktop example shows a shared counter reaching exactly 2000 with its built-in lock, and a block of four bytes read back through a second handle.

Try it yourself

1
Predict the picklable ones
Before running the second example, guess for each of the three candidates whether it pickles. Then add ("int", 5), ("open file", open("tools.py")) and a dict to the list and check your guesses. Plain data pickles; things tied to the operating system do not.
2
Run the Pool on your computer
Save the third example as pool_demo.py and run python pool_demo.py. Then delete the if __name__ == "__main__": line (un-indenting the body) and run it again to see what the missing guard does under spawn. Put the guard back afterwards.
3
Change the start method
Replace "spawn" with "forkserver" or "fork" (where your system has them) and note that the output is the same: the start method changes how workers begin, not what the program computes.

Code & diagrams

A process cannot start here, but pickling can be studied New tab

OSError is the browser sandbox saying it cannot create processes. The rest shows the isolation: the worker gets a copy.

Expected output

OSError: [Errno 52] Function not implemented
parent's list: ['a', 'b']
worker's reply: ['a', 'b', 'worker was here']

What can and cannot be pickled New tab

The program writes a small module, tools.py, and imports it, because pickle finds functions by module and name. Print the error type names only: the messages contain memory addresses.

Expected output

same function after the round trip: True
partial works: 1024
lambda -> PicklingError
local function -> AttributeError
lock -> TypeError
original: ['prompt 1', 'prompt 2']
copy:     ['prompt 1', 'prompt 2', 'added in the copy']

Pool.map and isolated memory (run on a normal Python)python

Not runnable in the browser. Verified with Python 3.12 on Windows, using the spawn context so it behaves the same on every system. The parent's counter stays 0.

import multiprocessing as mp

counter = 0


def bump(n):
    global counter
    counter += n               # changes this worker's own copy only
    return counter


def square(n):
    return n * n


if __name__ == "__main__":
    ctx = mp.get_context("spawn")
    with ctx.Pool(processes=2) as pool:
        print("map:", pool.map(square, range(6)))
        pool.map(bump, [1, 2, 3])
    print("parent's counter:", counter)

Output

map: [0, 1, 4, 9, 16, 25]
parent's counter: 0

Shared Value and SharedMemory (run on a normal Python)python

Not runnable in the browser. Verified with Python 3.12 on Windows. Value carries its own lock (get_lock), so two processes adding 1000 each give exactly 2000.

from multiprocessing import Process, Value, shared_memory


def add_many(counter, times):
    for _ in range(times):
        with counter.get_lock():
            counter.value += 1


if __name__ == "__main__":
    counter = Value("i", 0)            # a number in shared memory
    procs = [Process(target=add_many, args=(counter, 1000)) for _ in range(2)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    print("shared counter:", counter.value)

    shm = shared_memory.SharedMemory(create=True, size=4)
    shm.buf[:4] = bytes([1, 2, 3, 4])
    other = shared_memory.SharedMemory(name=shm.name)   # attach by name
    print("read through second handle:", list(other.buf[:4]))
    other.close()
    shm.close()
    shm.unlink()

Output

shared counter: 2000
read through second handle: [1, 2, 3, 4]

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Send a lambda to a Pool

Call pool.map with a lambda instead of a named function.

main.pywhole filepython

import multiprocessing as mp

if __name__ == "__main__":
    with mp.get_context("spawn").Pool(2) as pool:
        print(pool.map(lambda n: n * 2, [1, 2, 3]))

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

...

_pickle.PicklingError: Can't pickle <function <lambda> at 0x...>: attribute lookup <lambda> on __main__ failed

Myth vs fact

Myth

multiprocessing shares my variables between processes.

Fact

Each process has its own copy of memory. Changes in a worker are invisible to the parent unless you send them back or use shared objects.

Myth

Using processes is always faster than one process.

Fact

Starting workers and pickling data cost time. For small tasks the overhead is bigger than the saving.

Myth

The main guard is just a style habit.

Fact

Under spawn, it prevents a loop of processes starting processes. Without it the program raises a RuntimeError, or on some systems keeps starting workers.

Myth

fork is the default everywhere.

Fact

Not on Windows or macOS (spawn), and not on Linux from Python 3.14 (forkserver). Do not rely on fork behaviour.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Forking a process that already has threads copies only the calling thread, and any lock held by another thread stays locked forever in the child. This is why Python 3.12 started warning about os.fork() in multithreaded programs and why the Linux default moved away from fork in 3.14. If you must fork, do it before starting threads.
▸
The cost model: spawn startup is typically tens to hundreds of milliseconds per worker (a new interpreter plus imports), so create the pool once and reuse it, rather than a new pool per task. Pickle cost grows with data size; for big NumPy arrays, share memory or pass file paths and let each worker load its slice.
▸
Never unpickle data from an untrusted source: pickle can execute code while loading. Between your own processes it is fine; for anything coming from a network or a user, use JSON or another data-only format.
▸
Worker processes of PyTorch's DataLoader (num_workers), many scikit-learn tools and Gunicorn-style servers are all this machinery. When they misbehave on Windows or macOS, a missing main guard or an unpicklable object is the first thing to check.
▸
with Pool(...) as pool: calls pool.terminate() when the block ends, not close() plus join(). Anything still running is killed, so collect results (map, or get() on every apply_async result) inside the block. Also: a worker that raises sends the exception back to the parent, which re-raises it from map/get with the worker's formatted traceback attached as the cause, and a worker killed by the operating system (for example out of memory) can make a plain Pool hang, whereas ProcessPoolExecutor raises BrokenProcessPool.
▸
Joining a process that has put items on a multiprocessing.Queue which nobody has read can deadlock, because the child will not exit until its buffered data is flushed into the pipe. Drain the queue before join(). Pool worker processes are daemonic and cannot start child processes of their own.

Remember this

1
A process is a running program with its own memory. A multiprocessing.Process(target=fn, args=(...)) starts a new Python interpreter that runs fn. Because every process has its own GIL (Topic 12.2), pure-Python calculation in several processes truly runs in parallel on several cores. This is the standard answer to CPU-bound Python.
2
Memory is not shared. If a worker changes a global variable or a list, only its own copy changes; the parent never sees it. To get information back you return values (a Pool does this for you), put them in a multiprocessing.Queue, or use special shared objects. The isolation is a feature: no shared mutable state means no races.
3
Arguments and return values travel between processes as pickles: Python converts the object to bytes, sends them, and the other side rebuilds a copy. So every argument must be picklable. Plain data (numbers, strings, lists, dicts, dataclasses from Topic 8.8) pickles well. Lambdas, functions defined inside other functions, open files, locks and database connections do not. A top-level function is pickled by name, so the worker must be able to import it.
4
multiprocessing.Pool(processes=n) is the easy interface: pool.map(fn, items) splits the items among the workers and returns the results in order; pool.imap_unordered yields them as they finish. concurrent.futures.ProcessPoolExecutor (Topic 12.4) wraps the same machinery with the Future API and is usually the nicer choice for new code.
5
**The if __name__ == "__main__": guard is required. (Recall from Topic 6.2: that line runs its block only when the file is started directly, not when it is imported.) On Windows and macOS, new processes are started with spawn**: a fresh interpreter that imports your script to find the function to run. Without the guard the import would start more processes, which start more, so Python stops you with a RuntimeError. The older fork start method copies the parent's memory instead. It is fast but unsafe when the parent has threads. Linux used fork by default up to Python 3.13, and from 3.14 the Linux default is forkserver. macOS has used spawn since 3.8. Choose explicitly with multiprocessing.get_context("spawn") when you want the same behaviour everywhere.
6
For big data, copying costs time. multiprocessing.shared_memory.SharedMemory creates a block of memory that several processes attach to by name, and Value/Array hold small shared numbers. NumPy arrays can be built on top of shared memory. Shared data brings races back (use their locks), so share only what is large and read-mostly. In the AI world, heavy parallel work is usually left to libraries (NumPy, PyTorch, data loaders with worker processes) rather than hand-written processes.

Explain it without notes

How is a process different from a thread, and why does multiprocessing get around the GIL?

Why must arguments to a worker process be picklable, and which common things are not?

What is the if __name__ == "__main__": guard for in multiprocessing code?

Explain spawn, fork and forkserver, and say which defaults apply.

A function running inside pool.map raises ValueError. What does the parent program see, and how does the rest of the pool behave?

Practice

Write a top-level function cube(n) and show that pickle.loads(pickle.dumps(cube)) returns the same function. (In a script run as python file.py.)

Use functools.partial to make a picklable version of lambda x: pow(x, 3) and call it on 4 after a pickle round trip.

Write a function is_picklable(obj) that returns True or False, and test it on a list and on a threading.Lock().

Trade-offs

↔
Processes vs threads: real parallelism and isolation versus startup cost and copying of data.
↔
Pickling data vs shared memory: pickling is simple and safe but copies; shared memory avoids the copy but brings back locks and lifetime management.
↔
spawn vs fork: spawn is portable and safe but slower to start; fork is quick but unsafe with threads and not available everywhere.

Done when you can

Done when you can explain why a process pool can use several cores and a thread pool cannot for pure Python.
Done when you can say what must be true about an argument to a worker (it is picklable).
Done when you can write a Pool program with the main guard that works under spawn.
Done when you can name the default start method on Windows, macOS and Linux (including 3.14).
Done when you can say when shared memory is worth its complexity.

Command Palette

Words you'll meet

Step by step

01Separate kitchens, separate memory

02What a worker actually receives

03What can be pickled

04Pool.map and the main guard

05spawn, fork and forkserver

06Sharing on purpose

Try it yourself

Predict the picklable ones

Run the Pool on your computer

Change the start method

Code & diagrams

Break it on purpose

Send a lambda to a Pool

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can