osmontero Investigating Performance
Overview
Performance work fails in a specific way: the engineer reads the code, forms an intuition about what must be slow, optimizes that, and ships a change that measurably does nothing. The bottleneck is somewhere else, because the bottleneck is almost always somewhere else.
Core principle: Measure, localize, change one thing, measure again. Intuition selects what to measure; it never selects what to fix.
Companion skills: systematic-debugging — same discipline applied to wrongness rather than slowness. Both refuse to act before the cause is located.
The Iron Law
NO OPTIMIZATION WITHOUT A MEASUREMENT THAT NAMES THE BOTTLENECK
You must be able to say "X accounts for N% of the time" before you change X. Not "X looks expensive." Not "X is O(n²)." A measurement, with a number.
No exceptions:
- Not for "this is obviously the slow part" — obvious is what makes it worth measuring
- Not for "the fix is cheap anyway" — a cheap fix to a non-bottleneck is pure risk with zero payoff
- Not for changes you would make on style grounds regardless. Make them; just don't call it performance work
The corollary: a change without an after-measurement is not an optimization, it is a guess that shipped.
Step 0: Define the Target
"Slow" is not a bug report. Write down all four before touching anything:
| Example | |
|---|---|
| Metric | End-to-end latency of POST /documents |
| Statistic | p99, not mean — means hide the problem users complain about |
| Load | 200 concurrent clients, production-shaped data volume |
| Target | Under 300ms, from 2.1s today |
Without a target you cannot tell when you are done, and performance work has no natural stopping point. Without a statistic you will optimize the average and never touch the tail that generated the complaint.
Step 1: Reproduce It
You need a harness that produces the slow behavior on demand, repeatably.
- Real data shape. A thousand dev rows will not reproduce a missing index; ten million will.
- Warm up first, then measure. JIT compilation, connection pools, and page cache all make the first run a different program.
- Repeat and take the distribution. A single timing is noise. Report min/p50/p99, not one number.
- Pin the environment. Same machine, same build flags, no other load. Especially: measure a production build — a dev-mode framework build can be 10× slower and is not the thing you are shipping.
If you cannot reproduce it locally, measure in the environment where it happens. Production profiling at low sample rates is safe and is better than a faithful reproduction you do not have.
Step 2: Characterize Before Profiling
Two minutes here saves an hour of profiling the wrong dimension.
| Observation | What it implicates |
|---|---|
| Slow on the first call, fast after | Cold cache, lazy init, connection setup, JIT |
| Slow always, uniformly | An algorithm or a per-call fixed cost |
| Fast at low load, cliff at high load | Contention, pool exhaustion, or a queue — not the code path |
| Time scales with input size superlinearly | Complexity bug — find the nested loop before you profile |
| p50 fine, p99 terrible | GC pauses, lock convoy, retries, a rare slow path |
| CPU at 100% | Compute-bound → CPU profile |
| CPU low, latency high | Waiting — I/O, network, lock, or a sleep |
| Memory grows and never returns | Retention, not allocation rate → heap profile with two snapshots |
| Slow only in production | Data volume, concurrency, network topology, or config |
A quick sanity check: concurrency ≈ throughput × latency (Little's Law). If you serve 100 req/s at 2s each, roughly 200 requests are in flight. If the pool holds 20 connections, the pool is the bottleneck and no amount of query tuning will help.
Step 3: Profile the Right Dimension
Profile the dimension the characterization implicated. A CPU profile of an I/O-bound program shows an idle process and teaches you nothing.
| Dimension | When | What it shows |
|---|---|---|
| CPU | CPU pegged, compute-bound | Which functions burn cycles |
| Allocation | High GC time, memory churn | Where garbage comes from |
| Heap (in-use) | Memory grows without bound | What is retained, and by whom |
| Block / wait | Low CPU, high latency | What the program waits on |
| Mutex / lock | Degrades with concurrency | Contended locks |
| Execution trace | Latency spikes with no obvious owner | GC pauses, scheduler stalls, serialization |
| Query plan | The database is involved at all | Scans, bad join order, missing indexes |
Exact invocations per stack — Go, Node, Python, browser, SQL, system-level: references/profiling-tools.md.
Read the profile top-down by cumulative time, and be suspicious of flat profiles. A profile with no peak usually means the cost is spread across a call made far too many times — count the calls, don't just time them.
Step 4: The Usual Suspects
Once localized, the cause is usually on this list.
| Layer | Cause | Tell |
|---|---|---|
| Database | N+1 queries | Query count scales with result count |
| Database | Missing index | Seq Scan on a large table in the plan |
| Database | Index exists but is unused | Type mismatch, function on the column, or a leading-wildcard LIKE |
| Database | SELECT * over wide rows |
Bytes transferred dwarf the rows needed |
| Application | Work inside a loop that could be hoisted | Same call, same arguments, n times |
| Application | Serialization/deserialization | JSON encode/decode dominates a CPU profile |
| Application | Unbounded concurrency | Goroutine/promise count grows with load; memory follows |
| Application | Lock held across I/O | Mutex profile shows a long hold, not many acquisitions |
| Application | Allocation in a hot path | Alloc profile dominated by one call site |
| Network | Chatty protocol | Many small round trips where one batch would do |
| Network | No connection reuse | TLS handshakes in the trace |
| Frontend | Render-blocking waterfall | Sequential dependent requests in the network panel |
| Frontend | Work on the main thread | Long tasks in the performance trace |
| Frontend | Shipping the whole bundle | Large initial JS payload, most of it unused |
Before optimizing, ask whether the work can be avoided entirely. Caching, batching, pagination, doing it asynchronously, or not doing it are all bigger wins than making it faster, and usually simpler.
Step 5: Change One Thing, Then Measure Again
one change → re-measure → keep it or revert it → repeat
- One variable at a time. Two changes at once and you cannot attribute the result — including the case where one helped and the other hurt by the same amount.
- Revert what does not measurably help. Every optimization costs readability; unpaid-for complexity is a permanent tax.
- Confirm the improvement is real, not noise. Compare distributions across repeated runs (
benchstatand equivalents do this properly), not two single numbers. - Commit the benchmark with the fix. It is the only thing that will notice when the regression comes back.
- Re-profile after each win. The bottleneck moves. The second-biggest cost is now the biggest, and it is often somewhere entirely new.
Stop when you hit the target from Step 0. Continuing past it trades real maintainability for a number nobody asked for.
Tail Latency
If the complaint is about p99, the mean is actively misleading — a fix that improves the mean often makes the tail worse.
Common causes, in rough order of frequency:
- Queueing. Utilization above ~70% makes wait time rise sharply. The fix is capacity or backpressure, not faster code.
- GC pauses. Look at pause distribution, not total GC time. Reduce allocation rate or retained set.
- Lock convoy. One slow holder makes every waiter slow; latency spikes correlate with contention, not load.
- Retries. A timeout plus a retry turns one slow request into three, exactly when the system is least able to serve them.
- Cold paths. Cache misses, first-request-after-idle, connection re-establishment.
- Head-of-line blocking. One slow item stalls everything behind it in the same connection, batch, or partition.
Watch for coordinated omission in your load generator. A harness that waits for each response before sending the next request stops sending load exactly when the system is slowest — which erases the tail you are trying to measure. Use a generator that sends at a fixed rate, or one that explicitly corrects for it.
Red Flags
- "This is obviously the slow part" — with no profile
- Optimizing a function that a profile shows at 0.4% of runtime
- Benchmarking a dev build, or with an empty database
- A single timing used as evidence
- Two changes in one commit, both labeled performance
- A micro-benchmark improving while the end-to-end metric does not move
- Caching added before the cost of the uncached path was measured
- "It should be faster now"
All of these mean: go back and measure.
Common Mistakes
Optimizing the part you understand. Familiarity is not evidence. The bottleneck is disproportionately in code you did not write.
Micro-benchmarks that do not reflect real use. Amdahl's law is unforgiving: making 5% of the runtime twice as fast buys 2.5%.
Confusing allocation with retention. High allocation rate is a GC-pressure problem. Growing in-use heap is a leak. They need different profiles and different fixes.
Caching to hide a bug. A cache in front of an accidentally-quadratic function makes the problem invisible until the cache misses under exactly the conditions where you need it most.
Ignoring variance. A change that improves p50 and degrades p99 is usually a regression, because the tail is what users report.
No after-measurement. Without it you have shipped a diff, not a fix — and you have no idea which it was.