Python Performance Optimization Skill
Find and fix real performance problems. Measure first.
Load resources/implementation-playbook.md only for a named profiler recipe (cProfile, py-spy, tracemalloc, scalene). Do not load it for process or policy.
When to Use
- Slow Python, high latency, CPU, or memory
- Data-processing, I/O, or DB access that is measurably expensive
- Before/after profiling of a change
When Not to Use
- No evidence of a performance problem
- Quality KPIs (accuracy, F1, confidence) → algorithm-optimization
- Premature optimization off the critical path
- Non-Python performance
Related Skills
- Use python-concurrency when the bottleneck is I/O-bound or needs CPU parallelism.
- Use pyo3-maturin when a confirmed CPU hotspot cannot be accelerated further in Python.
- Use algorithm-optimization for quality KPIs on real cases, not runtime.
Core Principles
- Measure first — never optimize from guesses.
- Fix the actual hotspot, not the whole codebase.
- Simple readable fixes before rewrites.
- Re-measure after each meaningful change.
- Stop when the target is met.
Optimization Process
1. Clarify Goals
Latency, CPU, memory, or throughput? Numeric target? Constraints (Python version, deps, architecture freeze)?
2. Profile
- CPU / runtime:
cProfile, py-spy, scalene
- Memory:
tracemalloc, scalene (memory_profiler if already in the repo)
- Line-level:
line_profiler or sampling profilers
- I/O: timing/logs or async-aware profilers
Use time.perf_counter / timeit, not time.time(). Ignore micro-gains off the critical path.
3. Classify
| Type |
Signs |
Direction |
| CPU-bound |
High CPU, pure compute |
Algorithm, vectorization (NumPy/Polars if dataframes), concurrency |
| I/O-bound |
Waiting on net/disk/DB |
Async, batching, caching, pooling |
| Memory-bound |
High RSS, GC pressure |
Generators, in-place ops, smaller structures |
| Database |
Slow queries, N+1 |
Indexes, query shape, batching |
| Algorithmic |
Scales poorly with n |
Better complexity, early exit |
4. Apply (in order)
- Better algorithm or data structure
- Less work (cache, batch, skip repeats)
- Better libraries (Polars/NumPy where the repo already processes tables/arrays)
- Concurrency only when appropriate
- Low-level tricks last
5. Validate
Same benchmark as baseline. Correctness preserved. Gain worth the complexity.
Output Expectations
- Bottleneck statement with profiler evidence
- Prioritized changes
- Before/after when possible
- Trade-offs
Final Checklist
1---2name: python-performance3description: Profile and fix Python runtime, memory, or I/O bottlenecks with evidence (cProfile, py-spy, scalene). Not for accuracy/F1/quality KPI loops or unmeasured micro-optimizations.4---56# Python Performance Optimization Skill78Find and fix real performance problems. Measure first.910Load `resources/implementation-playbook.md` **only** for a named profiler recipe (cProfile, py-spy, tracemalloc, scalene). Do not load it for process or policy.1112## When to Use13- Slow Python, high latency, CPU, or memory14- Data-processing, I/O, or DB access that is measurably expensive15- Before/after profiling of a change1617## When Not to Use18- No evidence of a performance problem19- Quality KPIs (accuracy, F1, confidence) → **algorithm-optimization**20- Premature optimization off the critical path21- Non-Python performance2223## Related Skills24- Use **python-concurrency** when the bottleneck is I/O-bound or needs CPU parallelism.25- Use **pyo3-maturin** when a confirmed CPU hotspot cannot be accelerated further in Python.26- Use **algorithm-optimization** for quality KPIs on real cases, not runtime.2728## Core Principles291. Measure first — never optimize from guesses.302. Fix the actual hotspot, not the whole codebase.313. Simple readable fixes before rewrites.324. Re-measure after each meaningful change.335. Stop when the target is met.3435## Optimization Process3637### 1. Clarify Goals38Latency, CPU, memory, or throughput? Numeric target? Constraints (Python version, deps, architecture freeze)?3940### 2. Profile41- **CPU / runtime**: `cProfile`, `py-spy`, `scalene`42- **Memory**: `tracemalloc`, `scalene` (`memory_profiler` if already in the repo)43- **Line-level**: `line_profiler` or sampling profilers44- **I/O**: timing/logs or async-aware profilers4546Use `time.perf_counter` / `timeit`, not `time.time()`. Ignore micro-gains off the critical path.4748### 3. Classify4950| Type | Signs | Direction |51| :--- | :--- | :--- |52| **CPU-bound** | High CPU, pure compute | Algorithm, vectorization (NumPy/Polars if dataframes), concurrency |53| **I/O-bound** | Waiting on net/disk/DB | Async, batching, caching, pooling |54| **Memory-bound** | High RSS, GC pressure | Generators, in-place ops, smaller structures |55| **Database** | Slow queries, N+1 | Indexes, query shape, batching |56| **Algorithmic** | Scales poorly with n | Better complexity, early exit |5758### 4. Apply (in order)591. Better algorithm or data structure602. Less work (cache, batch, skip repeats)613. Better libraries (Polars/NumPy where the repo already processes tables/arrays)624. Concurrency only when appropriate635. Low-level tricks last6465### 5. Validate66Same benchmark as baseline. Correctness preserved. Gain worth the complexity.6768## Output Expectations69- Bottleneck statement with profiler evidence70- Prioritized changes71- Before/after when possible72- Trade-offs7374## Final Checklist75- [ ] Goal and constraints clear76- [ ] Profiled before changing77- [ ] Bottleneck type named78- [ ] Changes target the hotspot79- [ ] Re-measured80- [ ] Correctness preserved
dmoreq