Python Parallelism

Python Parallelism

Overview

Python's concurrency landscape changed fundamentally in 2025-2026. The GIL is becoming optional (Python 3.13+ experimental, 3.14 officially supported with 5-10% single-thread penalty). But for most code, the right tool depends on whether your workload is I/O-bound or CPU-bound — not whether the GIL exists.

Decision Framework

Is your workload I/O-bound or CPU-bound?

I/O-bound (network, disk, database):
  ├── Many concurrent connections? → asyncio (TaskGroup)
  ├── Simple parallel I/O? → ThreadPoolExecutor
  └── Background jobs? → Celery / Dramatiq / arq

CPU-bound (computation, data processing):
  ├── NumPy/pandas-style? → Polars (auto multi-core)
  ├── Custom Python code? → ProcessPoolExecutor
  ├── Large-scale distributed? → Dask or Ray
  └── Simple loop parallelism? → joblib

Mixed:
  └── asyncio + ProcessPoolExecutor (run_in_executor)

asyncio (I/O-Bound — Primary Choice)

TaskGroup (Python 3.11+) — Preferred Over gather()

async with asyncio.TaskGroup() as tg:
    task1 = tg.create_task(fetch_url(url1))
    task2 = tg.create_task(fetch_url(url2))
    task3 = tg.create_task(fetch_url(url3))
# All tasks complete here. If any raises, all are cancelled.
results = [task1.result(), task2.result(), task3.result()]

TaskGroup vs gather():

• TaskGroup: structured concurrency, automatic cancellation on failure, cleaner error handling
• gather(): returns results in order, return_exceptions=True for mixed success/failure
• Use TaskGroup for new code. Use gather() only when you need return_exceptions=True

Rate Limiting with Semaphore

sem = asyncio.Semaphore(10)  # Max 10 concurrent

async def limited_fetch(url):
    async with sem:
        return await fetch_url(url)

Common Pitfalls

Async Libraries

multiprocessing (CPU-Bound)

ProcessPoolExecutor (Recommended)

from concurrent.futures import ProcessPoolExecutor
import os

with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
    results = list(executor.map(cpu_intensive_task, data_chunks))

Pool Sizing

Start Method (Python 3.14 Change)

Python 3.14 changed the default start method for ProcessPoolExecutor to forkserver on Linux (was fork). This is safer but slightly slower to start.

Rules:

• Never mix fork with threads (deadlock risk)
• spawn is safest (default on Windows/macOS)
• forkserver is the new Linux default — good balance of safety and performance
• All data passed to workers must be picklable

Shared Memory (Large Data)

from multiprocessing import shared_memory
import numpy as np

shm = shared_memory.SharedMemory(create=True, size=array.nbytes)
shared_array = np.ndarray(array.shape, dtype=array.dtype, buffer=shm.buf)
shared_array[:] = array[:]  # Copy data once
# Workers access via shm.name — no serialization overhead

concurrent.futures

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=20) as executor:
    futures = {executor.submit(fetch, url): url for url in urls}
    for future in as_completed(futures):
        url = futures[future]
        try:
            result = future.result(timeout=30)
        except Exception as e:
            logger.error(f"Failed {url}: {e}")

submit() vs map():

• submit(): returns Future objects, use with as_completed() for results as they arrive
• map(): simpler, returns results in input order, less control

Data Processing Parallelism

2026 recommendation: For new data processing, default to Polars unless you need pandas compatibility. Polars automatically uses all CPU cores and has lazy evaluation.

Task Queues (Production Background Jobs)

Flask integration: Use Celery or Dramatiq with Redis broker. For simple cases, concurrent.futures in a background thread may suffice.

Retry Pattern

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=60),
    reraise=True
)
def call_external_api(url):
    response = httpx.get(url, timeout=10)
    response.raise_for_status()
    return response.json()

GIL-Free Python (3.13-3.14)

• Python 3.13: experimental --disable-gil build flag
• Python 3.14: officially supported free-threading with 5-10% single-thread penalty
• concurrent.interpreters (PEP 734): sub-interpreters with separate GILs — true parallelism without process overhead
• Don't rush to free-threading: Most C extensions aren't thread-safe yet. asyncio + multiprocessing covers 95% of use cases today

Connection Pooling

Rule: Always use connection pools for external resources. Creating new connections per request is a performance killer.

Anti-Patterns