Go Perf Methodology

Go Performance Methodology

"You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is." — Rob Pike, Notes on Programming in C

The high-level loop — profile or benchmark → change one thing → measure again — belongs to go-performance; read it first. This skill is its deep peer. It owns the layer beneath the loop: which diagnostic answers which question (and which is blind to which problem), Brendan Gregg's USE method as a saturation discipline, the statistical method that makes an A/B trustworthy, when a microbenchmark lies, the order of operations on a real slow system, reproducible measurement environments, CI regression gating, and tail-latency pitfalls. Everything else routes to a go-perf-* leaf.

All Go below builds clean under gofmt, go vet, and go test on Go 1.26.

1. The Tool-Selection Matrix — Which Diagnostic, Which Question

The single most expensive methodology error after "optimizing without measuring" is measuring with the wrong tool. The Go diagnostics page partitions diagnostics into four groups — profiling, tracing, debugging, and runtime statistics — and warns they are not interchangeable (Diagnostics). The decision layer:

Two rules carry this table:

• A CPU profile is blind to off-CPU time. It "determines where a program spends its time while actively consuming CPU cycles (as opposed to while sleeping or waiting for I/O)" (Diagnostics). A request that is slow because it is blocked on a lock, a channel, or a syscall contributes nothing to the CPU profile. Latency/blocking questions go to the tracer and block/mutex profiles, not pprof.
• The tracer is not a profiler. It is for understanding "how your goroutines execute … core runtime events such as GC runs … poorly parallelized execution," but "it is not great for identifying hot spots such as analyzing the cause of excessive memory or CPU usage. Use profiling tools instead first" (Diagnostics).

block and mutex profiles record nothing until enabled — "not enabled by default" — via runtime.SetBlockProfileRate and runtime.SetMutexProfileFraction (Diagnostics). Asking for a contention profile without setting the rate gets you an empty profile, not "no contention."

2. Profiles Are Samples — Read Them as Estimates

A profile is itself a statistical sample with known blind spots, not ground truth. When CPU profiling is on, "the Go program stops about 100 times per second and records a sample consisting of the program counters on the currently executing goroutine's stack" (Profiling Go Programs). The memory profiler "only records information for approximately one block per half megabyte allocated (the '1-in-524288 sampling rate'), so these are approximations" (pprof).

Consequences for how you read a profile:

• Profile long enough to accumulate samples. At ~100 Hz, a function with <10 ms total runtime may not appear at all; a 50 ms benchmark gives you ~5 samples — too few to trust. Short benchmarks make untrustworthy profiles.
• Flat vs cumulative. top shows flat time ("the number of samples in which the function was running, as opposed to waiting for a called function to return"); add -cum to "sort by … cumulative" and find expensive callers up the tree (pprof). Reading flat time when you want cumulative (or vice versa) misattributes the cost.
• Stacks truncate at 100 frames — deep recursion can be under-attributed (pprof).

Subcommand depth (list, peek, disasm, -diff_base, labels) → go-perf-pprof-profiling.

3. The USE Method Applied to Go

Before optimizing a function, ask which resource is the bottleneck. Brendan Gregg's USE method checks three things for every resource: Utilization ("the average time that the resource was busy servicing work"), Saturation ("the degree to which the resource has extra work which it can't service, often queued"), and Errors ("the count of error events"). The central instruction: "For every resource, check utilization, saturation, and errors" — a methodology that "solves about 80% of server issues with 5% of the effort" (Gregg, The USE Method). A resource includes software resources like "mutex locks, thread pools, and process capacity" (Gregg).

The Go translation — each resource to its diagnostic:

The decisive distinction: high utilization is not the problem; saturation is. A CPU pinned at 100% with an empty run queue is healthy; one at 60% with a deep run queue or lock queue is the real bottleneck — exactly the off-CPU case a CPU profile cannot see (§1). USE forces "is it CPU, memory/GC, locks, or the scheduler?" before any code change, so you don't CPU-optimize a memory- or lock-bound service.

4. Statistical Rigor — Why One Run Lies

go-performance says "use benchstat"; here is why, conceptually (the commands and flags are go-perf-benchmarking-statistics). benchstat "computes statistical summaries and A/B comparisons of Go benchmarks," and "each benchmark should be run at least 10 times to gather a statistically significant sample of results" (benchstat).

Encode/format=json-48   1.718µ ± 1%   1.423µ ± 1%  -17.20% (p=0.000 n=10)
Encode/format=gob-48    3.066µ ± 0%   3.070µ ± 2%        ~ (p=0.446 n=10)

Five principles below the one-line "use benchstat":

1. n=1 has no statistical power. A single ns/op pair cannot separate a real delta from CPU-frequency, GC, and scheduling noise. The test needs a sample to compute a p-value at all.
2. The test is non-parametric. By default benchstat "uses non-parametric statistics: median for summaries, and the Mann–Whitney U-test for A/B comparisons" — and "the p-value measures how likely it is that any differences were due to random chance (i.e., noise)" (benchstat).
3. ~ is a result, not a failure. It "means benchstat did not detect a statistically significant difference" (benchstat) — i.e. don't ship the complexity, the change bought nothing measurable.
4. Median, not mean. Benchmark noise is one-sided (interference only ever makes a run slower), so the median resists the outliers that skew a mean. Across a suite, the "geometric mean" row gives "an overall picture of how the benchmarks changed" — proportional, not an arithmetic average of percentages (benchstat).
5. α=0.05 means ~1 in 20 false positives. benchstat "uses an ɑ threshold of 0.05, which means it is expected to show a difference 5% of the time even if there is no difference" (benchstat). In a large table, a lone "significant" row may be chance — don't chase it.

5. Micro vs Macro — When a Benchmark Lies

A microbenchmark is a hypothesis generator, not proof of a production win: "Microbenchmarks help but full system testing is best" (go-perfbook). The ways one misleads:

• Cache-hot. The bench reuses one tiny dataset that lives in L1; production touches cold memory. "It's very easy for lookup tables to be far away in memory … making it faster to just recompute a value" (go-perfbook) — and the reverse: a cache-hot win evaporates on cold data.
• GC-quiet / branch-predicted. A tight single-allocation loop lets the GC and branch predictor settle into a best case that never occurs under mixed production load.
• Unrealistic input. "Match real-world input distributions; different data patterns provoke different algorithm behaviors" — e.g. "quicksort is O(n²) when data is sorted," and "if your expected input range is <100, your benchmarks should reflect that" (go-perfbook).
• No contention. A single-goroutine bench can't see the lock contention or false sharing that only appears at GOMAXPROCS cores under load.
• Dead-code elimination. The compiler deletes work whose result is unused (the sink trap — mechanics in go-testing-advanced; the methodology warning lives here).

Confirm a microbenchmark win with a macro/system benchmark or a production profile before claiming it: "some issues not apparent on your laptop might be visible once deployed" at "250k reqs/second on a 40 core server" (go-perfbook).

6. The Order of Operations

Optimization is "refactoring … [that] improves performance … generally … at the cost of readability" (go-perfbook) — so the bar to start is a measured goal you're missing, and the order is fixed (go-perfbook):

1. Set performance goals and confirm you're not meeting them. "The fastest code is the code that's never run" — first ask whether the work is needed at all.
2. Profile to find the real cost (CPU, allocations, blocking) — §1's matrix.
3. Benchmark with representative workloads and reproducible numbers (§5, §7).
4. Profile again to verify the issue is resolved.
5. Confirm significance with benchstat (§4) — ~ means it didn't work.
6. Verify the latency numbers make sense for the use case (§8).

Spend effort top-down: algorithm/data structure first, Go-specific implementation tweaks second, unsafe/assembly last. "Data dominates. If you've chosen the right data structures … the algorithms will almost always be self-evident" (Pike). Algorithmic wins dwarf micro-tuning — go-perfbook cites a ~30,000× algorithmic gain (1991–2008) against compilers merely doubling every 18 years (Proebsting's Law) (go-perfbook).

Amdahl's law is the cold-path guard. "Doubling speed in a routine consuming 5% of runtime yields only 2.5% total improvement. Focus on bottlenecks consuming 80%+ of runtime" (go-perfbook). A 50% speedup on a path that runs 0.1% of the time is noise you paid clarity for. The counterweight to Knuth's "premature optimization" is premature pessimization — don't "write code that is slower than it needs to be … when equivalently complex code would be faster" (go-perfbook); the rule is "don't micro-tune unmeasured," not "never think about performance."

7. A Reproducible Measurement Environment

The benchmark machine is itself a noise source; an unpinned machine makes §4's statistics meaningless. The dominant culprit is frequency scaling — turbo boost and thermal throttling make the same code benchmark 20%+ differently run-to-run.

perflock addresses both noise sources: it "is a simple locking wrapper for running benchmarks on shared hosts," acquiring "a system-wide lock while running command" (perflock). Two facts matter:

• Exclusive mode is the default — it "prevents any other perflock'd command from running," which serializes benchmarks so two never contend for CPU/cache at once (perflock).
• -governor defaults to 90% — it sets "CPU frequency to percent between the min and max while running command," pinning the clock below turbo so frequency is constant, and restores the governor on release (perflock).

Other controls (commands → go-perf-benchmarking-statistics / go-perf-os-tooling): pin the CPU governor to performance and disable turbo; taskset/isolcpus to keep the OS scheduler off the bench cores; fix GOMAXPROCS explicitly — and note Go 1.25 container-aware GOMAXPROCS changes the default under a cgroup CPU limit, so verify the floor when benching in a container (Go 1.25); quiesce the machine; prefer bare metal over a noisy-neighbor VM. Re-baseline after a Go upgrade — Green Tea GC is default in 1.26 and shifts the GC baseline (Go 1.26). Without governor pinning + serialization + n≥10, the delta is inside the noise floor and the U-test has nothing to bite on.

8. Tail Latency & Coordinated Omission

For a latency-sensitive service, report p99/p999, not the mean — and beware the measurement trap. Coordinated omission (Gil Tene, How NOT to Measure Latency): a load generator that waits for each request to finish before sending the next coordinates with the system under test — when the server stalls for 5 s, the generator sends nothing during the stall, recording one bad outlier instead of the hundreds of requests that should have been issued and would each show multi-second latency. The result is "catastrophically optimistic the moment the system under test starts to struggle." The fix is intended-start-time correction (measure latency from when a request should have started): tools wrk2 (constant rate -R + HdrHistogram) and Vegeta do this.

In Go, read /sched/pauses/total/gc:seconds (the GC stop-the-world pause distribution; /gc/pauses:seconds is its deprecated alias) and /sched/latencies:seconds from runtime/metrics. The dominant tail lever is usually lowering the allocation rate (which lowers GC frequency), ahead of any GC knob — Go's STW pauses are sub-millisecond and rarely dominate the tail. Depth → go-perf-tail-latency.

9. CI Regression Gating & Performance Budgets

A regression that isn't gated lands silently. Wire a job that runs the suite -count≥10 on the PR vs its base on the same runner, pipes both to benchstat, and fails the build when a tracked benchmark regresses past a threshold (p<0.05 by the U-test, not eyeballing). Principles:

• Gate allocations, not just time. allocs/op is deterministic per code path, far less noisy than ns/op, so an alloc-count budget is the cheapest, most reliable gate — a jump from 1→3 allocs/op is a real regression even when timing is noisy.
• Budgets are goals made executable ("p99 < 50 ms", "0 allocs/op on the hot encode path"). go-perfbook's step 1 — "determine performance goals; confirm you're not meeting them" — turns "is this fast enough?" from opinion into a CI assertion (go-perfbook).
• Prefer relative A/B over absolute floors. The CI runner is shared and noisy (§7), so absolute ns/op thresholds are fragile; compare PR vs base in the same job and treat ~ as "no regression."

10. Don't

• Don't reach for a CPU profile to explain latency or blocking. It is blind to off-CPU waits; use the tracer + block/mutex profiles (Diagnostics).
• Don't run multiple diagnostics at once and trust them all. "Use tools in isolation … precise memory profiling skews CPU profiles" and goroutine-blocking profiling perturbs the scheduler trace (Diagnostics).
• Don't trust one benchmark run. n=1 has no power; use n≥10 + benchstat, and accept ~ as "no win" (benchstat).
• Don't optimize a cold path for a big percentage. Amdahl: 2× on 5% of runtime = 2.5% total (go-perfbook).
• Don't confuse "fast in a microbenchmark" with "fast in production" — confirm on a macro bench or a production profile (§5).
• Don't benchmark on an unpinned machine. Turbo and noisy neighbors bury any real delta; serialize and pin the clock (perflock) (perflock).
• Don't report the mean for a latency SLO, and don't let the load generator coordinate-omit the tail (§8).
• Don't tune GOGC/GOMEMLIMIT before USE confirms GC is the bottleneck — the usual fix is fewer allocations (§3, §8).

11. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-benchmarking-statistics — benchstat flags, p-values, geomean, sample size, perflock how-to, A/B method, CI gating commands.
• go-perf-pprof-profiling — CPU/heap pprof subcommands, inuse vs alloc, differential profiling, labels, production profiling.
• go-perf-block-mutex-profiles — block/mutex/goroutine profiles and the enablement gotcha.
• go-perf-execution-tracer — runtime/trace, go tool trace, the flight recorder (1.25).
• go-perf-escape-analysis — reading -gcflags=-m; what escapes to the heap.
• go-perf-gc-tuning — GOGC/GOMEMLIMIT, gctrace, allocation rate as the primary lever.
• go-perf-tail-latency — p99/p999, coordinated omission, latency histograms.
• go-perf-os-tooling — Linux perf, flame graphs, perflock as a benchmark stabilizer.

Parent / sibling marketplace (go-* correctness):

• go-performance — the measure-first overview and the 3-step loop this skill deepens.
• go-testing-advanced — benchmark mechanics: for b.Loop(), b.ReportAllocs, b.ResetTimer, b.RunParallel, the sink/DCE trap.
• go-idiomatic-discipline — the policy root; unmeasured micro-opt vs clarity is its axis-2 ("clear AND correct") failure.
• go-version-feature-map — version floors: container-aware GOMAXPROCS 1.25, Green Tea GC default 1.26.

12. Reference Files

Methodology anti-patterns in LLM-generated Go performance work, each with wrong/right contrast and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml