Python Performance

Python Performance Patterns

Profiling Tools

# CPU profiling
python -m cProfile -s cumtime -o profile.out script.py
python -m pstats profile.out  # interactive viewer
snakeviz profile.out          # visual flamegraph (pip install snakeviz)

# Line profiler (most useful for identifying hot lines)
pip install line_profiler
kernprof -l -v script.py  # requires @profile decorator

# Memory profiler
pip install memory_profiler
python -m memory_profiler script.py  # requires @profile decorator
mprof run script.py && mprof plot    # memory over time

Profiling in Code

import cProfile
import pstats
import io
from contextlib import contextmanager

@contextmanager
def profile_block(n_top: int = 20):
    pr = cProfile.Profile()
    pr.enable()
    yield
    pr.disable()
    s = io.StringIO()
    ps = pstats.Stats(pr, stream=s).sort_stats("cumulative")
    ps.print_stats(n_top)
    print(s.getvalue())

with profile_block():
    result = expensive_computation()

Key Optimizations

Use __slots__ for hot objects

@dataclass
class Point:
    __slots__ = ("x", "y")  # 3x less memory, faster attribute access
    x: float
    y: float

Avoid global lookups in tight loops

# Bad: each iteration looks up `math.sqrt` in global namespace
import math
for x in big_list:
    result = math.sqrt(x)

# Good: local binding
from math import sqrt
for x in big_list:
    result = sqrt(x)

Use built-ins and stdlib over hand-rolled code

# Sorting
sorted_items = sorted(items, key=lambda x: x.score, reverse=True)

# Grouping
from itertools import groupby
for key, group in groupby(sorted(items, key=attrgetter("category")), key=attrgetter("category")):
    ...

# Counting
from collections import Counter
counts = Counter(item.category for item in items)

Numpy for numeric work

import numpy as np

# Bad: Python loop for numeric computation
result = [x * 2 + 1 for x in large_list]  # slow

# Good: vectorized numpy
arr = np.array(large_list)
result = arr * 2 + 1  # 100x faster for large arrays

Numba for JIT compilation

from numba import jit, njit

@njit  # no-python mode: compiles to machine code
def compute_distances(points: np.ndarray) -> np.ndarray:
    n = len(points)
    distances = np.zeros((n, n))
    for i in range(n):
        for j in range(i + 1, n):
            d = np.sqrt(((points[i] - points[j]) ** 2).sum())
            distances[i, j] = distances[j, i] = d
    return distances

Common Bottlenecks

• String concatenation in loops → use "".join(parts)
• in on lists with large sets → convert to set first
• Repeated dict.get() / attribute access → local binding
• JSON parsing in loops → batch or cache
• Missing database indexes → check EXPLAIN ANALYZE