Performance Profiling Skill
Optimization Golden Rule
"Don't optimize without a baseline."
Always measure -> change -> measure.
Critical Metrics (The 4 Golden Signals)
- Latency: Time it takes to serve a request. (p50, p95, p99)
- Traffic: Demand on your system (req/sec).
- Errors: Rate of requests that fail.
- Saturation: How "full" your service is (CPU/Memory usage).
Profiling Tools & Techniques
Python
Node.js
Database (SQL)
Frontend (Browser)
- Lighthouse: Core Web Vitals (LCP, CLS, INP).
- Chrome Performance Tab: Main thread blocking time.
- Network Waterfall: Time to First Byte (TTFB).
Optimization Hierarchy
- Database/IO: (Indexing, Caching, Batching) - Biggest Gains
- Algorithm: (O(n²) -> O(n log n))
- Memory: (Allocation churn, GC pressure)
- Micro-optimization: (Loop unrolling, etc.) - Smallest Gains
Common Rationalizations
| Excuse |
Why It's Wrong |
| "It feels slow, let me optimize this function" |
Feelings aren't data — profile first, then optimize the actual bottleneck |
| "We should optimize everything" |
Premature optimization is the root of all evil — focus on the critical path |
| "Caching will fix it" |
Caching masks problems and adds complexity — fix the root cause first |
| "It's fast enough in dev" |
Dev has 1 user — production has thousands and cold caches |
| "We'll optimize later" |
Performance debt compounds — a 100ms regression per sprint = 5s in a year |
Example
# Capture a 30-second CPU flamegraph from a running Python service
py-spy record -o profile.svg --duration 30 --pid "$(pgrep -f my-service)"
# Identify top 3 hot functions
py-spy top --pid "$(pgrep -f my-service)"
Then, per the optimization hierarchy, start with DB/IO fixes (indexing, batching, caching the right layer) before touching algorithm-level changes.
Rules
- MUST capture a baseline measurement before proposing any change
- NEVER optimize code without profiler data pointing at it as the bottleneck
- CRITICAL: report p95/p99, not just p50 — averages hide real user pain
- MANDATORY: follow the hierarchy — DB/IO before algorithm before micro-optimization
Gotchas
py-spy needs CAP_SYS_PTRACE on Linux and SIP-disabled codesigning on macOS to attach to another process. Containerized services usually run without ptrace privileges — profiling requires a --cap-add=SYS_PTRACE on the container or an in-process alternative (cProfile, yappi).- Production hosts frequently set
/proc/sys/kernel/perf_event_paranoid=2 or higher, which disables user-space perf events. Tools that rely on perf (perf, bcc, bpftrace) silently produce empty output — check cat /proc/sys/kernel/perf_event_paranoid first.
- Node.js
--prof output gets interleaved across worker threads and child processes. A single isolate-*.log mixes samples from multiple isolates unless each worker writes its own — filter by PID or use clinic flame which handles the split.
- Chrome DevTools samples at ~1kHz; operations faster than ~1ms vanish. For microbenchmarks, prefer
performance.now() with manual markers, not the Performance tab.
EXPLAIN ANALYZE on Postgres executes the query, including INSERT/UPDATE/DELETE — wrap write queries in a transaction that you roll back, or use EXPLAIN (ANALYZE, BUFFERS) ... ; ROLLBACK; in one statement.
When NOT to Use
- For correctness bugs (wrong output) — use
/debug
- For frontend render bugs without timing data — measure with DevTools first
- For infrastructure capacity planning — use load testing, not profiling
- For generic code quality — use
/analyze
1---2name: performance-profiling3description: "Performance: golden signals, p50/p95/p99, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, bottleneck, memory leak."4effort: medium5allowed-tools: Read, Grep6user-invocable: false7---89# Performance Profiling Skill1011## Optimization Golden Rule12**"Don't optimize without a baseline."**13Always measure -> change -> measure.1415## Critical Metrics (The 4 Golden Signals)161. **Latency**: Time it takes to serve a request. (p50, p95, p99)172. **Traffic**: Demand on your system (req/sec).183. **Errors**: Rate of requests that fail.194. **Saturation**: How "full" your service is (CPU/Memory usage).2021## Profiling Tools & Techniques2223### Python24- **CPU Sampling**: `py-spy`25 ```bash26 # Record flamegraph27 py-spy record -o profile.svg --pid <pid>28 ```29- **Function Profiling**: `cProfile`30 ```python31 import cProfile32 cProfile.run('main()')33 ```3435### Node.js36- **Flamegraphs**: `0x` or built-in profiler.37 ```bash38 node --prof app.js39 node --prof-process isolate-0xnnnnn.log > processed.txt40 ```41- **Event Loop**: `clinic doctor`4243### Database (SQL)44- **Explain Plan**: Analyze query cost.45 ```sql46 EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM users WHERE active = 1;47 ```48- **N+1 Problem**: Look for loop-generated queries.4950### Frontend (Browser)51- **Lighthouse**: Core Web Vitals (LCP, CLS, INP).52- **Chrome Performance Tab**: Main thread blocking time.53- **Network Waterfall**: Time to First Byte (TTFB).5455## Optimization Hierarchy561. **Database/IO**: (Indexing, Caching, Batching) - *Biggest Gains*572. **Algorithm**: (O(n²) -> O(n log n))583. **Memory**: (Allocation churn, GC pressure)594. **Micro-optimization**: (Loop unrolling, etc.) - *Smallest Gains*6061## Common Rationalizations6263| Excuse | Why It's Wrong |64|--------|----------------|65| "It feels slow, let me optimize this function" | Feelings aren't data — profile first, then optimize the actual bottleneck |66| "We should optimize everything" | Premature optimization is the root of all evil — focus on the critical path |67| "Caching will fix it" | Caching masks problems and adds complexity — fix the root cause first |68| "It's fast enough in dev" | Dev has 1 user — production has thousands and cold caches |69| "We'll optimize later" | Performance debt compounds — a 100ms regression per sprint = 5s in a year |7071## Example7273```bash74# Capture a 30-second CPU flamegraph from a running Python service75py-spy record -o profile.svg --duration 30 --pid "$(pgrep -f my-service)"7677# Identify top 3 hot functions78py-spy top --pid "$(pgrep -f my-service)"79```8081Then, per the optimization hierarchy, start with DB/IO fixes (indexing, batching, caching the right layer) before touching algorithm-level changes.8283## Rules8485- **MUST** capture a baseline measurement before proposing any change86- **NEVER** optimize code without profiler data pointing at it as the bottleneck87- **CRITICAL**: report p95/p99, not just p50 — averages hide real user pain88- **MANDATORY**: follow the hierarchy — DB/IO before algorithm before micro-optimization8990## Gotchas9192- `py-spy` needs `CAP_SYS_PTRACE` on Linux and SIP-disabled codesigning on macOS to attach to another process. Containerized services usually run without ptrace privileges — profiling requires a `--cap-add=SYS_PTRACE` on the container or an in-process alternative (`cProfile`, `yappi`).93- Production hosts frequently set `/proc/sys/kernel/perf_event_paranoid=2` or higher, which disables user-space perf events. Tools that rely on perf (`perf`, `bcc`, `bpftrace`) silently produce empty output — check `cat /proc/sys/kernel/perf_event_paranoid` first.94- Node.js `--prof` output gets interleaved across worker threads and child processes. A single `isolate-*.log` mixes samples from multiple isolates unless each worker writes its own — filter by PID or use `clinic flame` which handles the split.95- Chrome DevTools samples at ~1kHz; operations faster than ~1ms vanish. For microbenchmarks, prefer `performance.now()` with manual markers, not the Performance tab.96- `EXPLAIN ANALYZE` on Postgres **executes** the query, including `INSERT`/`UPDATE`/`DELETE` — wrap write queries in a transaction that you roll back, or use `EXPLAIN (ANALYZE, BUFFERS) ... ; ROLLBACK;` in one statement.9798## When NOT to Use99100- For correctness bugs (wrong output) — use `/debug`101- For frontend render bugs without timing data — measure with DevTools first102- For infrastructure capacity planning — use load testing, not profiling103- For generic code quality — use `/analyze`
softspark