Python Multiprocessing

With the Python multiprocessing library, we can write truly parallel software. When we were using Python threads, we weren’t utilizing multiple CPUs or CPU cores. However, with this library, we will. To start, we need to import the multiprocessing module instead of threading.

Multiprocessing based example

Here’s the multiprocessing version of the code we used in the baseline version and the threaded version:

import time
import multiprocessing

# A CPU heavy calculation, just
# as an example. This can be
# anything you like
def heavy(n, myid):
    for x in range(1, n):
        for y in range(1, n):
            x**y
    print(myid, "is done")

def multiproc(n):
    processes = []

    for i in range(n):
        p = multiprocessing.Process(target=heavy, args=(500,i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

if __name__ == "__main__":
    start = time.time()
    multiproc(80)
    end = time.time()
    print("Took: ", end - start)Code language: Python (python)

As you can see, this looks almost the same as the threaded version, code-wise. The threading and multiprocessing modules are intentionally made very equivalent. But this time, the 80 invocations of heavy run on all CPU cores at the same time.

On my laptop, with 12 CPU cores and Python 3.15, the baseline version takes 31 seconds. The threaded version takes just as long, because of the GIL. This multiprocessing version finishes in under 5 seconds, more than six times faster.

Your numbers will differ, but the pattern is the same: for CPU-bound code, the speedup grows with the number of CPU cores. It’s not perfectly linear, though. Starting processes takes time, and on many modern CPUs some cores are slower, energy-efficient cores. My laptop has 8 fast cores and 4 efficient ones, which is why I get a factor of six and not twelve.

Why you need if __name__ == "__main__"

You might wonder why the code that starts the processes is inside an if __name__ == "__main__": block. It has to do with how Python starts a new process. There are three start methods:

• spawn starts a fresh Python interpreter, which imports your script and then runs the function you gave it. This is the default on Windows and macOS.
• forkserver starts a small server process once, which then creates (forks) the worker processes. This is the default on Linux since Python 3.14.
• fork copies the running process. It was the default on Linux before Python 3.14, but it can cause hard-to-find problems when your program also uses threads.

With spawn and forkserver, every new process imports your script. Without the if __name__ == "__main__": guard, each child process would start new processes itself when it imports the script. Python detects this and stops with an error. So always put the code that starts processes behind this guard.

For the same reason, the function and its arguments are sent to the new process by pickling them. Stick to functions defined at the top level of a module and to simple arguments like numbers, strings, lists and dictionaries. A lambda, for example, can’t be pickled, so you can’t use it as a target.

Python multiprocessing pool

We can make the multiprocessing version a little more elegant and slightly faster by using multiprocessing.Pool(p). This Python multiprocessing helper creates a pool of size p processes. If you don’t supply a value for p, it will default to the number of CPU cores available to your program, which is a sensible choice most of the time.

By using the Pool.map() method, we can submit work to the pool. This work comes in the form of a simple function call:

import time
import multiprocessing

# A CPU heavy calculation, just
# as an example. This can be
# anything you like
def heavy(n, myid):
    for x in range(1, n):
        for y in range(1, n):
            x**y
    print(myid, "is done")

def doit(n):
    heavy(500, n)

def pooled(n):
    # By default, our pool will have
    # one process per CPU core
    with multiprocessing.Pool() as pool:
        pool.map(doit, range(n))

if __name__ == "__main__":
    start = time.time()
    pooled(80)
    end = time.time()
    print("Took: ", end - start)Code language: Python (python)

This version’s runtime is roughly the same as the non-pooled version (4.5 seconds on my laptop), but it has to create fewer processes, so it is more efficient. After all, instead of creating 80 processes, it creates one per CPU core (12 on my laptop) and reuses those for all 80 jobs.

ProcessPoolExecutor: the high-level alternative

The standard library has a second, higher-level way to do this: ProcessPoolExecutor from the concurrent.futures module. For new code, this is the API I recommend. You submit work and get back a future, an object that will hold the result once it’s ready. Exceptions in a worker are raised again in your main program when you ask for the result, so errors don’t get lost:

import time
from concurrent.futures import ProcessPoolExecutor

# A CPU heavy calculation, just
# as an example. This can be
# anything you like
def heavy(n, myid):
    for x in range(1, n):
        for y in range(1, n):
            x**y
    return f"{myid} is done"

def pooled(n):
    with ProcessPoolExecutor() as executor:
        futures = [executor.submit(heavy, 500, i) for i in range(n)]
        for future in futures:
            print(future.result())

if __name__ == "__main__":
    start = time.time()
    pooled(80)
    end = time.time()
    print("Took: ", end - start)Code language: Python (python)

This one took 4.1 seconds on my laptop, about the same as the pool. A nice bonus: concurrent.futures also has a ThreadPoolExecutor with the same interface, so switching between threads and processes is a one-word change. There’s also executor.map(), which works like Pool.map().

What about free-threaded Python?

Since Python 3.13, there’s an optional free-threaded build of Python without the GIL. In that build, regular threads can run Python code on multiple CPU cores at the same time, without the overhead of separate processes. It’s still optional: the standard Python you download uses the GIL, and not all third-party packages support the free-threaded build yet. For now, multiprocessing is the reliable way to use all your cores. If you want to experiment, read the official free-threading HOWTO.

Keep learning

If you want to learn more about Python multiprocessing and working with multiprocessing pools, you can head over to the official documentation. If you work on the command line a lot, you might be interested in my article on Bash multiprocessing as well!