Python Profiler

Python Profiler Skill

Overview

Comprehensive Python profiling skill that integrates multiple profiling tools to identify performance bottlenecks, memory leaks, and optimization opportunities in Python applications.

Capabilities

1. CPU Profiling

• cProfile: Built-in profiler for function-level analysis
• Pyinstrument: Visual call stack with flame graphs
• py-spy: Low-overhead sampling profiler for production
• line_profiler: Line-by-line performance analysis

2. Memory Profiling

• memory_profiler: Track memory usage line-by-line
• Scalene: Combined CPU + memory profiling
• Pyroscope: Continuous real-time profiling

3. Concurrency Profiling

• Yappi: Profile multithreading and greenlets/coroutines
• Thread-safe profiling for asyncio applications

Usage

Basic CPU Profiling

# Using cProfile
python -m cProfile -o output.prof your_script.py

# Using Pyinstrument
pyinstrument --html your_script.py

Memory Analysis

# Using memory_profiler
from memory_profiler import profile

@profile
def my_function():
    # Your code here
    pass

Production Profiling

# Using py-spy (no code changes needed)
py-spy record -o profile.svg --pid <process_id>

Integration Scripts

detect_bottleneck.py

Automatically runs multiple profilers and generates comprehensive reports:

#!/usr/bin/env python3
import cProfile
import pstats
from pyinstrument import Profiler

def profile_function(func):
    """Profile a function with multiple tools"""
    # cProfile analysis
    profiler = cProfile.Profile()
    profiler.enable()
    result = func()
    profiler.disable()

    stats = pstats.Stats(profiler)
    stats.sort_stats('cumulative')
    stats.print_stats(20)

    return result

Best Practices

1. Start with cProfile for high-level overview
2. Use Pyinstrument to visualize call hierarchies
3. Apply line_profiler to specific bottleneck functions
4. Monitor memory with memory_profiler for large datasets
5. Use py-spy in production (no overhead, no code changes)

Output Interpretation

• cumtime: Total time in function + subcalls
• tottime: Time in function excluding subcalls
• ncalls: Number of times function was called
• percall: Average time per call

Requirements

pip install cProfile pyinstrument memory-profiler py-spy line-profiler scalene yappi

Example Workflow

1. Run cProfile to identify slow functions
2. Visualize with Pyinstrument flame graph
3. Deep-dive with line_profiler on specific functions
4. Check memory usage with memory_profiler
5. Optimize and re-profile to confirm improvements

Common Bottlenecks to Look For

• Nested loops with high iteration counts
• Inefficient list comprehensions
• Repeated string concatenations
• Unoptimized pandas operations
• Missing database query optimizations (N+1 queries)
• Synchronous I/O in async code