Performance Optimization

Performance Optimization Skill

Apply these principles when optimizing code for performance. Focus on the critical 3% where performance truly matters - a 12% improvement is never marginal in engineering.

Core Philosophy

1. Measure First: Never optimize without profiling data
2. Estimate Costs: Back-of-envelope calculations before implementation
3. Avoid Work: The fastest code is code that doesn't run
4. Reduce Allocations: Memory allocation is often the hidden bottleneck
5. Cache Locality: Memory access patterns dominate modern performance

Reference Latency Numbers

Use these for back-of-envelope calculations:

Optimization Techniques (Priority Order)

1. Algorithmic Improvements (Highest Impact)

Always check algorithm complexity first:

O(N^2) → O(N log N)  = 1000x faster for N=1M
O(N)   → O(1)        = unbounded improvement

Common patterns:

• Replace nested loops with hash lookups
• Use sorted-list intersection → hash table lookups
• Process in reverse post-order to eliminate per-element checks
• Replace interval trees (O(log N)) with hash maps (O(1)) when ranges aren't needed

2. Reduce Memory Allocations

Allocation is expensive (~25-100ns + GC pressure)

# BAD: Allocates on every call
def process(items):
    result = []  # New allocation
    for item in items:
        result.append(transform(item))
    return result

# GOOD: Pre-allocate or reuse
def process(items, out=None):
    if out is None:
        out = [None] * len(items)
    for i, item in enumerate(items):
        out[i] = transform(item)
    return out

Techniques:

• Pre-size containers with reserve() or known capacity
• Hoist temporary containers outside loops
• Reuse buffers across iterations (clear instead of recreate)
• Move instead of copy large structures
• Use stack allocation for bounded-lifetime objects

3. Compact Data Structures

Minimize memory footprint and cache lines touched:

// BAD: 24 bytes due to padding
struct Item {
    flag: bool,      // 1 byte + 7 padding
    value: i64,      // 8 bytes
    count: i32,      // 4 bytes + 4 padding
}

// GOOD: 16 bytes with reordering
struct Item {
    value: i64,      // 8 bytes
    count: i32,      // 4 bytes
    flag: bool,      // 1 byte + 3 padding
}

Techniques:

• Reorder struct fields by size (largest first)
• Use smaller numeric types when range permits (u8 vs i32)
• Replace 64-bit pointers with 32-bit indices
• Keep hot fields together, move cold fields to separate struct
• Use bit vectors for small-domain sets
• Flatten nested maps: Map<A, Map<B, C>> → Map<(A,B), C>

4. Fast Paths for Common Cases

Optimize the common case without hurting rare cases:

# BAD: Always takes slow path
def parse_varint(data):
    return generic_varint_parser(data)

# GOOD: Fast path for common 1-byte case
def parse_varint(data):
    if data[0] < 128:  # Single byte - 90% of cases
        return data[0], 1
    return generic_varint_parser(data)  # Rare multi-byte

Techniques:

• Handle common dimensions inline (1-D, 2-D tensors)
• Check for empty/trivial inputs early
• Specialize for common sizes (small strings, few elements)
• Process trailing elements separately to avoid slow generic code

5. Precompute Expensive Information

Trade memory for compute when beneficial:

# BAD: Recomputes on every access
def is_vowel(char):
    return char.lower() in 'aeiou'

# GOOD: Lookup table
VOWEL_TABLE = [c.lower() in 'aeiou' for c in (chr(i) for i in range(256))]
def is_vowel(char):
    return VOWEL_TABLE[ord(char)]

Techniques:

• Precompute flags/properties at construction time
• Build lookup tables for character classification
• Cache expensive computed properties
• Validate at boundaries once, not repeatedly inside

6. Bulk/Batch APIs

Amortize fixed costs across multiple operations:

# BAD: N round trips
for item in items:
    result = db.lookup(item)

# GOOD: 1 round trip
results = db.lookup_many(items)

Design APIs that support:

• Batch lookups instead of individual fetches
• Vectorized operations over loops
• Streaming interfaces for large datasets

7. Avoid Unnecessary Work

# BAD: Always computes expensive value
def process(data, config):
    expensive = compute_expensive(data)  # Always runs
    if config.needs_expensive:
        use(expensive)

# GOOD: Defer until needed
def process(data, config):
    if config.needs_expensive:
        expensive = compute_expensive(data)  # Only when needed
        use(expensive)

Techniques:

• Lazy evaluation for expensive operations
• Short-circuit evaluation in conditions
• Move loop-invariant code outside loops
• Specialize instead of using general-purpose libraries in hot paths

8. Help the Compiler/Runtime

Lower-level optimizations when profiling shows need:

// Avoid function call overhead in hot loops
#[inline(always)]
fn hot_function(x: i32) -> i32 { x * 2 }

// Copy to local variable for better alias analysis
fn process(data: &mut [i32], factor: &i32) {
    let f = *factor;  // Compiler knows this won't change
    for x in data {
        *x *= f;
    }
}

Techniques:

• Use raw pointers/indices instead of iterators in critical loops
• Hand-unroll very hot loops (4+ iterations)
• Move slow-path code to separate non-inlined functions
• Avoid abstractions that hide costs in hot paths

9. Reduce Synchronization

Minimize lock contention and atomic operations:

• Default to thread-compatible (external sync) not thread-safe
• Sample statistics (1-in-32) instead of tracking everything
• Batch updates to shared state
• Use thread-local storage for per-thread data

Profiling Workflow

1. Identify hotspots: Use profiler (pprof, perf, py-spy)
2. Measure baseline: Write benchmark before optimizing
3. Estimate improvement: Calculate expected gain
4. Implement change: Focus on one optimization at a time
5. Verify improvement: Run benchmark, confirm gain
6. Check for regressions: Ensure no other code paths slowed down

When Profile is Flat (No Clear Hotspots)

• Pursue multiple small 1-2% improvements collectively
• Look for restructuring opportunities higher in call stack
• Profile allocations specifically (often hidden cost)
• Gather hardware performance counters (cache misses, branch mispredicts)
• Consider if the code is already well-optimized

Anti-Patterns to Avoid

1. Premature optimization in non-critical code
2. Optimizing without measurement - changes based on intuition
3. Micro-optimizing while ignoring algorithmic complexity
4. Copying when moving would suffice
5. Growing containers one element at a time (quadratic)
6. Allocating in loops when reuse is possible
7. String formatting in hot paths (use pre-built templates)
8. Regex when simple string matching suffices

Estimation Template

Before optimizing, estimate:

Operation cost: ___ ns/us/ms
Frequency: ___ times per second/request
Total time: cost × frequency = ___
Improvement target: ___% reduction
Expected new time: ___
Is this worth it? [ ] Yes [ ] No