Python Performance Optimization

Python Performance Optimization

Expert guidance for profiling, optimizing, and accelerating Python applications through systematic analysis, algorithmic improvements, efficient data structures, and acceleration techniques.

When to Use This Skill

• Code runs too slowly for production requirements
• High CPU usage or memory consumption issues
• Need to reduce API response times or batch processing duration
• Application fails to scale under load
• Optimizing data processing pipelines or scientific computing
• Reducing cloud infrastructure costs through efficiency gains
• Profile-guided optimization after measuring performance bottlenecks

Core Concepts

The Golden Rule: Never optimize without profiling first. 80% of execution time is spent in 20% of code.

Optimization Hierarchy (in priority order):

1. Algorithm complexity - O(n²) → O(n log n) provides exponential gains
2. Data structure choice - List → Set for lookups (10,000x faster)
3. Language features - Comprehensions, built-ins, generators
4. Caching - Memoization for repeated calculations
5. Compiled extensions - NumPy, Numba, Cython for hot paths
6. Parallelism - Multiprocessing for CPU-bound work

Key Principle: Algorithmic improvements beat micro-optimizations every time.

Quick Reference

Load detailed guides for specific optimization areas:

Optimization Workflow

Phase 1: Measure

1. Profile with cProfile - Identify slow functions
2. Line profile hot paths - Find exact slow lines
3. Memory profile - Check for memory bottlenecks
4. Benchmark baseline - Record current performance

Phase 2: Analyze

1. Check algorithm complexity - Is it O(n²) or worse?
2. Evaluate data structures - Are you using lists for lookups?
3. Identify repeated work - Can results be cached?
4. Find I/O bottlenecks - Database queries, file operations

Phase 3: Optimize

1. Improve algorithms first - Biggest impact
2. Use appropriate data structures - Set/dict for O(1) lookups
3. Apply caching - @lru_cache for expensive functions
4. Use generators - For large datasets
5. Leverage NumPy/Numba - For numerical code
6. Parallelize - Multiprocessing for CPU-bound tasks

Phase 4: Validate

1. Re-profile - Verify improvements
2. Benchmark - Measure speedup quantitatively
3. Test correctness - Ensure optimizations didn't break functionality
4. Document - Explain why optimization was needed

Common Optimization Patterns

Pattern 1: Replace List with Set for Lookups

# Slow: O(n) lookup
if item in large_list:  # Bad

# Fast: O(1) lookup
if item in large_set:   # Good

Pattern 2: Use Comprehensions

# Slower
result = []
for i in range(n):
    result.append(i * 2)

# Faster (35% speedup)
result = [i * 2 for i in range(n)]

Pattern 3: Cache Expensive Calculations

from functools import lru_cache

@lru_cache(maxsize=None)
def expensive_function(n):
    # Result cached automatically
    return complex_calculation(n)

Pattern 4: Use Generators for Large Data

# Memory inefficient
def read_file(path):
    return [line for line in open(path)]  # Loads entire file

# Memory efficient
def read_file(path):
    for line in open(path):  # Streams line by line
        yield line.strip()

Pattern 5: Vectorize with NumPy

# Pure Python: ~500ms
result = sum(i**2 for i in range(1000000))

# NumPy: ~5ms (100x faster)
import numpy as np
result = np.sum(np.arange(1000000)**2)

Common Mistakes to Avoid

1. Optimizing before profiling - You'll optimize the wrong code
2. Using lists for membership tests - Use sets/dicts instead
3. String concatenation in loops - Use "".join() or StringIO
4. Loading entire files into memory - Use generators
5. N+1 database queries - Use JOINs or batch queries
6. Ignoring built-in functions - They're C-optimized and fast
7. Premature optimization - Focus on algorithmic improvements first
8. Not benchmarking - Always measure improvements quantitatively

Decision Tree

Start here: Profile with cProfile to find bottlenecks

Hot path is algorithm?

• Yes → Check complexity, improve algorithm, use better data structures
• No → Continue

Hot path is computation?

• Numerical loops → NumPy or Numba
• CPU-bound → Multiprocessing
• Already fast enough → Done

Hot path is memory?

• Large data → Generators, streaming
• Many objects → __slots__, object pooling
• Caching needed → @lru_cache or custom cache

Hot path is I/O?

• Database → Batch queries, indexes, connection pooling
• Files → Buffering, streaming
• Network → Async I/O, request batching

Best Practices

1. Profile before optimizing - Measure to find real bottlenecks
2. Optimize algorithms first - O(n²) → O(n) beats micro-optimizations
3. Use appropriate data structures - Set/dict for lookups, not lists
4. Leverage built-ins - C-implemented built-ins are faster than pure Python
5. Avoid premature optimization - Optimize hot paths identified by profiling
6. Use generators for large data - Reduce memory usage with lazy evaluation
7. Batch operations - Minimize overhead from syscalls and network requests
8. Cache expensive computations - Use @lru_cache or custom caching
9. Consider NumPy/Numba - Vectorization and JIT for numerical code
10. Parallelize CPU-bound work - Use multiprocessing to utilize all cores

Resources

• Python Performance: https://wiki.python.org/moin/PythonSpeed
• cProfile: https://docs.python.org/3/library/profile.html
• NumPy: https://numpy.org/doc/stable/user/absolute_beginners.html
• Numba: https://numba.pydata.org/
• Cython: https://cython.readthedocs.io/
• High Performance Python (Book by Gorelick & Ozsvald)