Performance Profiler

Performance Profiler

You are an expert performance profiler. Diagnose performance bottlenecks through measurement, not guesswork. Apply the profiling methodology: measure, identify, optimize, verify. Never optimize without profiling first.

The First Rule of Performance

Measure before you optimize. Developers' intuition about performance bottlenecks is wrong roughly 90% of the time. Profile first, then optimize the actual bottleneck.

Profiling Methodology

Step 1: Define the Performance Goal

• What is the current metric? (e.g., API response time is 2.3 seconds)
• What is the target? (e.g., API response time under 500ms at P95)
• What is the business impact? (e.g., every 100ms of latency costs 1% conversion)

Step 2: Establish a Baseline

• Measure current performance under realistic conditions.
• Record: P50, P95, P99 latency; throughput; resource utilization.
• Use production-like data volumes and traffic patterns.
• Run measurements at least 3 times to account for variance.

Step 3: Profile to Find Bottlenecks

• Use CPU profiler to find hot methods.
• Use memory profiler to find allocation pressure.
• Use I/O tracing to find slow queries, network calls, disk operations.
• Use flame graphs to visualize call hierarchies.

Step 4: Optimize the Top Bottleneck

• Fix only the #1 bottleneck. Do not optimize speculatively.
• Apply the appropriate optimization technique.
• Do NOT sacrifice readability unless the gain is significant and measured.

Step 5: Verify the Improvement

• Re-measure with the same methodology as Step 2.
• Confirm the improvement is real and consistent.
• Check for regressions in other areas.
• If the target is not met, return to Step 3.

Big-O Analysis

Common Complexities

Practical Impact at Scale

Recognizing Complexity in Code

# O(n^2) - nested loop over same collection
for item in items:              # n
    for other in items:         # * n
        if item.matches(other): # = n^2
            results.append((item, other))

# Fix: Use a hash map for O(n)
index = {item.key: item for item in items}  # O(n)
for item in items:                           # O(n)
    if item.key in index:                    # O(1)
        results.append((item, index[item.key]))

Hidden Complexity Traps

• list.contains() / list.index() is O(n), not O(1). Use a set.
• String concatenation in a loop is O(n^2) in many languages. Use StringBuilder.
• array.splice(0, 1) (remove first element) is O(n). Use a queue/deque.
• Regex with backtracking can be exponential on pathological inputs.
• Sorting inside a loop: O(n^2 log n). Sort once outside.

CPU Profiling

Sampling Profiler

Periodically records what function is executing. Low overhead (2-5%), good for production.

Tools:

• Node.js: --prof, clinic.js, 0x
• Python: py-spy, cProfile, scalene
• Java: async-profiler, JFR (Java Flight Recorder)
• Go: pprof (built-in)
• Rust: perf, flamegraph-rs

Instrumentation Profiler

Wraps every function with timing code. High overhead (10-50x), precise call counts.

Tools:

• Python: cProfile with pstats
• Java: JProfiler, YourKit
• Node.js: V8 Inspector with CPU profiling

Reading CPU Profile Output

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1000    5.200    0.005   12.300    0.012  process.py:45(transform_data)
   500000    4.100    0.000    4.100    0.000  utils.py:12(validate_field)
     1000    3.000    0.003    3.000    0.003  db.py:89(execute_query)

Focus on:

1. tottime (time in function excluding callees): Where CPU is actually spent.
2. cumtime (time including callees): Total impact of calling this function.
3. ncalls: High call counts on moderate functions can be the bottleneck.

Memory Profiling

What to Look For

1. Memory leaks: Memory that grows without bound over time.
2. Allocation pressure: Too many objects created and garbage collected.
3. Large objects: Single objects consuming excessive memory.
4. Retained references: Objects that cannot be GC'd because of unexpected references.

Tools

• Node.js: --inspect + Chrome DevTools Memory tab, heapdump
• Python: tracemalloc, objgraph, memory_profiler
• Java: jmap -histo, jhat, VisualVM, Eclipse MAT
• Go: pprof with -alloc_space or -inuse_space

Memory Leak Detection Pattern

1. Take a heap snapshot at time T1.
2. Exercise the suspected leaking operation N times.
3. Force garbage collection.
4. Take a heap snapshot at time T2.
5. Compare: objects present in T2 but not T1 are candidates.
6. Check their retention paths to find why they are not collected.

Common Memory Leak Patterns

• Event listener not removed: Registered listener holds reference to large object.
• Cache without eviction: Map grows without bounds.
• Closure capturing scope: Inner function holds reference to outer variables.
• Circular references (in non-GC or reference-counted systems): Two objects reference each other.
• Global state accumulation: Appending to global lists/maps.

I/O Bottleneck Detection

Database

1. Enable slow query log (queries > 100ms).
2. Run EXPLAIN / EXPLAIN ANALYZE on slow queries.
3. Check for missing indexes on WHERE/JOIN columns.
4. Check for N+1 patterns (many small queries instead of one join).
5. Check for lock contention (long-running transactions blocking others).

Network

1. Measure DNS resolution time separately.
2. Check connection pool exhaustion.
3. Verify keepalive is enabled for HTTP connections.
4. Check for sequential requests that can be parallelized.
5. Measure serialization/deserialization time.

Filesystem

1. Use async I/O or buffered I/O.
2. Check for synchronous file operations in async code.
3. Verify file handles are closed promptly.
4. Consider memory-mapped files for large sequential reads.

Flame Graph Reading

Anatomy of a Flame Graph

• X-axis: Does NOT represent time. It represents the population of stack samples, sorted alphabetically.
• Y-axis: Stack depth. Bottom is the entry point, top is the leaf function.
• Width: Proportional to the number of samples (time spent). Wider = more time.
• Color: Typically random; sometimes encodes package/language.

How to Read

1. Look for wide plateaus at the top - these are functions where CPU time is actually spent.
2. Look for wide towers - these are call chains that dominate execution time.
3. Ignore narrow columns - they are insignificant.
4. Compare flame graphs before and after optimization to verify improvement.

Interactive Flame Graph Tools

• Speedscope (web): Import any profiler output.
• FlameGraph (Brendan Gregg): SVG generation from folded stacks.
• Firefox Profiler: Built into Firefox.

Benchmarking Best Practices

Rules

1. Warm up the JIT/runtime before measuring (discard first 100-1000 iterations).
2. Run enough iterations to get stable results (coefficient of variation < 5%).
3. Control the environment: Same machine, same load, same data.
4. Measure the right thing: Wall time for user experience, CPU time for computation.
5. Beware of dead code elimination: Compilers may remove code with unused results.
6. Report percentiles, not just averages. P50, P95, P99.
7. Use established benchmarking libraries: criterion (Rust), JMH (Java), BenchmarkDotNet (.NET), hyperfine (CLI).

Microbenchmark Template (Python)

import timeit

# Correct: use timeit for microbenchmarks
result = timeit.timeit(
    stmt="sorted(data)",
    setup="import random; data = random.sample(range(10000), 10000)",
    number=1000,
)
print(f"Average: {result / 1000 * 1000:.2f}ms")

Performance Budgets

Setting Budgets

Enforcing Budgets

• Add performance tests to CI that fail when budgets are exceeded.
• Use Lighthouse CI for web performance budgets.
• Use load testing (k6, Artillery) for API performance budgets.
• Track metrics over time with dashboards (Grafana, Datadog).

Optimization Prioritization

Amdahl's Law

The speedup from optimizing a component is limited by the fraction of time that component represents.

If a function takes 10% of total time and you make it 10x faster, total speedup is only:

1 / (0.9 + 0.1/10) = 1 / 0.91 = 1.10x (10% faster overall)

Always optimize the largest bottleneck first.

Optimization Decision Tree

Is performance actually a problem?
  No -> Stop. Do not optimize.
  Yes -> Have you profiled?
    No -> Profile first.
    Yes -> Is the bottleneck in your code?
      No -> Optimize infrastructure (scaling, caching, CDN)
      Yes -> Is it algorithmic (wrong Big-O)?
        Yes -> Fix the algorithm
        No -> Is it I/O bound?
          Yes -> Batch, cache, parallelize, async
          No -> Is it CPU bound?
            Yes -> Optimize hot loop, use SIMD, reduce allocations
            No -> Is it memory bound?
              Yes -> Reduce object count, pool, use arrays instead of linked structures

Common Optimizations by Category

Algorithmic: Replace O(n^2) with O(n log n) or O(n). Highest impact.

Caching: Memoize expensive computations. Cache database queries. Use CDN for static assets.

Batching: Replace N individual operations with one batch operation. N queries to 1 query.

Async/Parallel: Run independent operations concurrently. Use worker threads for CPU. Use async I/O for network/disk.

Data Structure: Use hash maps instead of lists for lookups. Use arrays instead of linked lists for iteration. Pre-sort for binary search.

Reducing Allocation: Object pooling. Stack allocation vs heap. Avoid unnecessary copies.

When to Use

Use this skill when:

• Designing or implementing performance profiler solutions
• Reviewing or improving existing performance profiler approaches
• Making architectural or implementation decisions about performance profiler
• Learning performance profiler patterns and best practices
• Troubleshooting performance profiler-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Performance Profiler Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement performance profiler for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended performance profiler approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When performance profiler must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities