# Go Perf Methodology > Guides Go performance work as a measure-first discipline AT DEPTH — the deep peer of go-performance. Owns the tool-selection matrix (which question a benchmark answers vs a CPU profile vs a heap profile's inuse/alloc vs the execution tracer vs block/mutex profiles vs GODEBUG=gctrace), Brendan Gregg's USE method mapped to Go resources, statistical rigor (why one run lies, benchstat ≥10 runs / p-value / geomean), micro-vs-macro benchmark design, the order of operations and Amdahl's law. Auto-invokes when planning a Go optimization, choosing a diagnostic, setting a perf budget, or judging whether a change is worth it, and on "is this worth optimizing", "which profiler should I use", "how do I know my optimization worked", "why does my benchmark say X but prod says Y". Routes the high-level loop to go-performance, benchmark mechanics to go-testing-advanced, benchstat details to go-perf-benchmarking-statistics. - Skill: `ctoth/go-perf-methodology` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add ctoth/go-perf-methodology` - Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-methodology/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Finance & Business - Author: ctoth (https://skillmd.com/u/ctoth) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ctoth/go-perf-methodology --- # Go Performance Methodology > "You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is." > — Rob Pike, [Notes on Programming in C](https://www.lysator.liu.se/c/pikestyle.html) The high-level loop — **profile or benchmark → change one thing → measure again** — belongs to **`go-performance`**; read it first. This skill is its **deep peer**. It owns the layer *beneath* the loop: **which** diagnostic answers **which** question (and which is blind to which problem), Brendan Gregg's USE method as a saturation discipline, the statistical method that makes an A/B trustworthy, when a microbenchmark lies, the order of operations on a real slow system, reproducible measurement environments, CI regression gating, and tail-latency pitfalls. Everything else routes to a `go-perf-*` leaf. All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.26. --- ## 1. The Tool-Selection Matrix — Which Diagnostic, Which Question The single most expensive *methodology* error after "optimizing without measuring" is **measuring with the wrong tool**. The Go diagnostics page partitions diagnostics into four groups — profiling, tracing, debugging, and runtime statistics — and warns they are not interchangeable ([Diagnostics](https://go.dev/doc/diagnostics)). The decision layer: | Question / symptom | Right diagnostic | Wrong tool (why) | Routes to | |---|---|---|---| | "Is this function faster / fewer allocs now?" | benchmark + benchstat, n≥10 | one run (inside the noise) | `go-perf-benchmarking-statistics` | | "Where does CPU time go?" | CPU profile (`-cpuprofile`) | tracer ("not great … for excessive CPU usage" [[Diag](https://go.dev/doc/diagnostics)]) | `go-perf-pprof-profiling` | | "What allocates / leaks?" | heap profile — `inuse_*` (live now) vs `alloc_*` (cumulative) | CPU profile | `go-perf-pprof-profiling` | | "Why slow *sometimes* / tail spikes / poor parallelism?" | execution tracer + flight recorder | CPU profile (blind to off-CPU waits) | `go-perf-execution-tracer` | | "Which lock or channel is contended?" | block / mutex profile (**enable first**) | CPU profile (off-CPU blind) | `go-perf-block-mutex-profiles` | | "Are goroutines piling up / leaking?" | goroutine profile (stack traces of all goroutines) | — | `go-perf-block-mutex-profiles` | | "Does this value escape to the heap?" | `go build -gcflags=-m` | heap profile alone | `go-perf-escape-analysis` | | "Is GC overhead the cost?" | `GODEBUG=gctrace=1`, `/gc/*` `runtime/metrics` | guessing at `GOGC` | `go-perf-gc-tuning` | | "Which resource is saturated at all?" | USE method (§3) | jumping to "optimize the function" | this skill | Two rules carry this table: - **A CPU profile is blind to off-CPU time.** It "determines where a program spends its time while actively consuming CPU cycles (as opposed to while sleeping or waiting for I/O)" ([Diagnostics](https://go.dev/doc/diagnostics)). A request that is slow because it is *blocked on a lock, a channel, or a syscall* contributes nothing to the CPU profile. Latency/blocking questions go to the tracer and block/mutex profiles, not pprof. - **The tracer is not a profiler.** It is for understanding "how your goroutines execute … core runtime events such as GC runs … poorly parallelized execution," but "it is not great for identifying hot spots such as analyzing the cause of excessive memory or CPU usage. Use profiling tools instead first" ([Diagnostics](https://go.dev/doc/diagnostics)). `block` and `mutex` profiles record **nothing** until enabled — "not enabled by default" — via `runtime.SetBlockProfileRate` and `runtime.SetMutexProfileFraction` ([Diagnostics](https://go.dev/doc/diagnostics)). Asking for a contention profile without setting the rate gets you an empty profile, not "no contention." --- ## 2. Profiles Are Samples — Read Them as Estimates A profile is itself a statistical sample with known blind spots, not ground truth. When CPU profiling is on, "the Go program stops about 100 times per second and records a sample consisting of the program counters on the currently executing goroutine's stack" ([Profiling Go Programs](https://go.dev/blog/pprof)). The memory profiler "only records information for approximately one block per half megabyte allocated (the '1-in-524288 sampling rate'), so these are approximations" ([pprof](https://go.dev/blog/pprof)). Consequences for how you read a profile: - **Profile long enough to accumulate samples.** At ~100 Hz, a function with <10 ms total runtime may not appear at all; a 50 ms benchmark gives you ~5 samples — too few to trust. Short benchmarks make untrustworthy profiles. - **Flat vs cumulative.** `top` shows flat time ("the number of samples in which the function was running, as opposed to waiting for a called function to return"); add `-cum` to "sort by … cumulative" and find expensive *callers* up the tree ([pprof](https://go.dev/blog/pprof)). Reading flat time when you want cumulative (or vice versa) misattributes the cost. - **Stacks truncate at 100 frames** — deep recursion can be under-attributed ([pprof](https://go.dev/blog/pprof)). Subcommand depth (`list`, `peek`, `disasm`, `-diff_base`, labels) → **`go-perf-pprof-profiling`**. --- ## 3. The USE Method Applied to Go Before optimizing *a function*, ask which *resource* is the bottleneck. Brendan Gregg's USE method checks three things **for every resource**: **Utilization** ("the average time that the resource was busy servicing work"), **Saturation** ("the degree to which the resource has extra work which it can't service, often queued"), and **Errors** ("the count of error events"). The central instruction: "For every resource, check utilization, saturation, and errors" — a methodology that "solves about 80% of server issues with 5% of the effort" ([Gregg, The USE Method](https://www.brendangregg.com/usemethod.html)). A resource includes **software resources** like "mutex locks, thread pools, and process capacity" ([Gregg](https://www.brendangregg.com/usemethod.html)). The Go translation — each resource to its diagnostic: | Resource | Utilization | Saturation (the tell) | Go tool | |---|---|---|---| | CPU | CPU% / `GOMAXPROCS` busy | runnable-queue depth | `GODEBUG=schedtrace`, CPU profile | | Memory | heap in-use bytes | GC frequency, swap, OOM | `runtime/metrics` `/gc/heap/*`, `gctrace` | | Scheduler / goroutines | P busy | `/sched/latencies:seconds` (runnable→running wait) | execution tracer, `schedtrace` | | Locks (software resource) | time held | block/mutex profile contention | mutex / block profile | | Channels / worker pool | workers busy | buffered-queue depth, blocked sends | tracer, block profile | | GC | GC CPU fraction | mark-assist time | `/sched/pauses/total/gc:seconds`, `gctrace` | The decisive distinction: **high utilization is not the problem; saturation is.** A CPU pinned at 100% with an empty run queue is healthy; one at 60% with a deep run queue or lock queue is the real bottleneck — exactly the off-CPU case a CPU profile cannot see (§1). USE forces "is it CPU, memory/GC, locks, or the scheduler?" *before* any code change, so you don't CPU-optimize a memory- or lock-bound service. --- ## 4. Statistical Rigor — Why One Run Lies `go-performance` says "use benchstat"; here is *why*, conceptually (the commands and flags are **`go-perf-benchmarking-statistics`**). benchstat "computes statistical summaries and A/B comparisons of Go benchmarks," and "each benchmark should be run at least 10 times to gather a statistically significant sample of results" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). ``` Encode/format=json-48 1.718µ ± 1% 1.423µ ± 1% -17.20% (p=0.000 n=10) Encode/format=gob-48 3.066µ ± 0% 3.070µ ± 2% ~ (p=0.446 n=10) ``` Five principles below the one-line "use benchstat": 1. **n=1 has no statistical power.** A single `ns/op` pair cannot separate a real delta from CPU-frequency, GC, and scheduling noise. The test *needs* a sample to compute a p-value at all. 2. **The test is non-parametric.** By default benchstat "uses non-parametric statistics: median for summaries, and the Mann–Whitney U-test for A/B comparisons" — and "the p-value measures how likely it is that any differences were due to random chance (i.e., noise)" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). 3. **`~` is a result, not a failure.** It "means benchstat did not detect a statistically significant difference" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)) — i.e. *don't ship the complexity*, the change bought nothing measurable. 4. **Median, not mean.** Benchmark noise is one-sided (interference only ever makes a run slower), so the median resists the outliers that skew a mean. Across a suite, the "geometric mean" row gives "an overall picture of how the benchmarks changed" — proportional, not an arithmetic average of percentages ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). 5. **α=0.05 means ~1 in 20 false positives.** benchstat "uses an ɑ threshold of 0.05, which means it is expected to show a difference 5% of the time even if there is no difference" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). In a large table, a lone "significant" row may be chance — don't chase it. --- ## 5. Micro vs Macro — When a Benchmark Lies A microbenchmark is a *hypothesis generator*, not proof of a production win: "Microbenchmarks help but full system testing is best" ([go-perfbook](https://github.com/dgryski/go-perfbook)). The ways one misleads: - **Cache-hot.** The bench reuses one tiny dataset that lives in L1; production touches cold memory. "It's very easy for lookup tables to be far away in memory … making it faster to just recompute a value" ([go-perfbook](https://github.com/dgryski/go-perfbook)) — and the reverse: a cache-hot win evaporates on cold data. - **GC-quiet / branch-predicted.** A tight single-allocation loop lets the GC and branch predictor settle into a best case that never occurs under mixed production load. - **Unrealistic input.** "Match real-world input distributions; different data patterns provoke different algorithm behaviors" — e.g. "quicksort is O(n²) when data is sorted," and "if your expected input range is <100, your benchmarks should reflect that" ([go-perfbook](https://github.com/dgryski/go-perfbook)). - **No contention.** A single-goroutine bench can't see the lock contention or false sharing that only appears at `GOMAXPROCS` cores under load. - **Dead-code elimination.** The compiler deletes work whose result is unused (the sink trap — mechanics in **`go-testing-advanced`**; the methodology warning lives here). Confirm a microbenchmark win with a macro/system benchmark or a production profile before claiming it: "some issues not apparent on your laptop might be visible once deployed" at "250k reqs/second on a 40 core server" ([go-perfbook](https://github.com/dgryski/go-perfbook)). --- ## 6. The Order of Operations Optimization is "refactoring … [that] improves performance … generally … at the cost of readability" ([go-perfbook](https://github.com/dgryski/go-perfbook)) — so the bar to start is a measured goal you're missing, and the order is fixed ([go-perfbook](https://github.com/dgryski/go-perfbook)): 1. **Set performance goals and confirm you're not meeting them.** "The fastest code is the code that's never run" — first ask whether the work is needed at all. 2. **Profile** to find the real cost (CPU, allocations, blocking) — §1's matrix. 3. **Benchmark** with representative workloads and reproducible numbers (§5, §7). 4. **Profile again** to verify the issue is resolved. 5. **Confirm significance with benchstat** (§4) — `~` means it didn't work. 6. **Verify the latency numbers make sense** for the use case (§8). Spend effort top-down: **algorithm/data structure first**, Go-specific implementation tweaks second, `unsafe`/assembly last. "Data dominates. If you've chosen the right data structures … the algorithms will almost always be self-evident" ([Pike](https://www.lysator.liu.se/c/pikestyle.html)). Algorithmic wins dwarf micro-tuning — go-perfbook cites a ~30,000× algorithmic gain (1991–2008) against compilers merely doubling every 18 years (Proebsting's Law) ([go-perfbook](https://github.com/dgryski/go-perfbook)). **Amdahl's law is the cold-path guard.** "Doubling speed in a routine consuming 5% of runtime yields only 2.5% total improvement. Focus on bottlenecks consuming 80%+ of runtime" ([go-perfbook](https://github.com/dgryski/go-perfbook)). A 50% speedup on a path that runs 0.1% of the time is noise you paid clarity for. The counterweight to Knuth's "premature optimization" is *premature pessimization* — don't "write code that is slower than it needs to be … when equivalently complex code would be faster" ([go-perfbook](https://github.com/dgryski/go-perfbook)); the rule is "don't *micro*-tune unmeasured," not "never think about performance." --- ## 7. A Reproducible Measurement Environment The benchmark machine is itself a noise source; an unpinned machine makes §4's statistics meaningless. The dominant culprit is **frequency scaling** — turbo boost and thermal throttling make the *same* code benchmark 20%+ differently run-to-run. **perflock** addresses both noise sources: it "is a simple locking wrapper for running benchmarks on shared hosts," acquiring "a system-wide lock while running command" ([perflock](https://github.com/aclements/perflock)). Two facts matter: - **Exclusive mode is the default** — it "prevents any other perflock'd command from running," which **serializes** benchmarks so two never contend for CPU/cache at once ([perflock](https://github.com/aclements/perflock)). - **`-governor` defaults to 90%** — it sets "CPU frequency to `percent` between the min and max while running command," pinning the clock below turbo so frequency is *constant*, and restores the governor on release ([perflock](https://github.com/aclements/perflock)). Other controls (commands → `go-perf-benchmarking-statistics` / `go-perf-os-tooling`): pin the CPU governor to `performance` and disable turbo; `taskset`/`isolcpus` to keep the OS scheduler off the bench cores; fix `GOMAXPROCS` explicitly — and note **Go 1.25 container-aware `GOMAXPROCS`** changes the default under a cgroup CPU limit, so verify the floor when benching in a container ([Go 1.25](https://go.dev/doc/go1.25)); quiesce the machine; prefer bare metal over a noisy-neighbor VM. Re-baseline after a Go upgrade — **Green Tea GC is default in 1.26** and shifts the GC baseline ([Go 1.26](https://go.dev/doc/go1.26)). Without governor pinning + serialization + n≥10, the delta is inside the noise floor and the U-test has nothing to bite on. --- ## 8. Tail Latency & Coordinated Omission For a latency-sensitive service, **report p99/p999, not the mean** — and beware the measurement trap. **Coordinated omission** (Gil Tene, [How NOT to Measure Latency](https://www.youtube.com/watch?v=lJ8ydIuPFeU)): a load generator that waits for each request to finish before sending the next *coordinates* with the system under test — when the server stalls for 5 s, the generator sends nothing during the stall, recording one bad outlier instead of the hundreds of requests that should have been issued and would each show multi-second latency. The result is "catastrophically optimistic the moment the system under test starts to struggle." The fix is **intended-start-time correction** (measure latency from when a request *should* have started): tools `wrk2` (constant rate `-R` + HdrHistogram) and Vegeta do this. In Go, read `/sched/pauses/total/gc:seconds` (the GC stop-the-world pause distribution; `/gc/pauses:seconds` is its deprecated alias) and `/sched/latencies:seconds` from `runtime/metrics`. The dominant tail lever is usually **lowering the allocation rate** (which lowers GC frequency), ahead of any GC knob — Go's STW pauses are sub-millisecond and rarely dominate the tail. Depth → **`go-perf-tail-latency`**. --- ## 9. CI Regression Gating & Performance Budgets A regression that isn't gated lands silently. Wire a job that runs the suite `-count≥10` on the PR vs its base **on the same runner**, pipes both to benchstat, and **fails the build when a tracked benchmark regresses past a threshold** (p<0.05 by the U-test, not eyeballing). Principles: - **Gate allocations, not just time.** `allocs/op` is deterministic per code path, far less noisy than `ns/op`, so an alloc-count budget is the cheapest, most reliable gate — a jump from 1→3 allocs/op is a real regression even when timing is noisy. - **Budgets are goals made executable** ("p99 < 50 ms", "0 allocs/op on the hot encode path"). go-perfbook's step 1 — "determine performance goals; confirm you're not meeting them" — turns "is this fast enough?" from opinion into a CI assertion ([go-perfbook](https://github.com/dgryski/go-perfbook)). - **Prefer relative A/B over absolute floors.** The CI runner is shared and noisy (§7), so absolute `ns/op` thresholds are fragile; compare PR vs base in the same job and treat `~` as "no regression." --- ## 10. Don't - **Don't reach for a CPU profile to explain latency or blocking.** It is blind to off-CPU waits; use the tracer + block/mutex profiles ([Diagnostics](https://go.dev/doc/diagnostics)). - **Don't run multiple diagnostics at once and trust them all.** "Use tools in isolation … precise memory profiling skews CPU profiles" and goroutine-blocking profiling perturbs the scheduler trace ([Diagnostics](https://go.dev/doc/diagnostics)). - **Don't trust one benchmark run.** n=1 has no power; use n≥10 + benchstat, and accept `~` as "no win" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). - **Don't optimize a cold path for a big percentage.** Amdahl: 2× on 5% of runtime = 2.5% total ([go-perfbook](https://github.com/dgryski/go-perfbook)). - **Don't confuse "fast in a microbenchmark" with "fast in production"** — confirm on a macro bench or a production profile (§5). - **Don't benchmark on an unpinned machine.** Turbo and noisy neighbors bury any real delta; serialize and pin the clock (perflock) ([perflock](https://github.com/aclements/perflock)). - **Don't report the mean for a latency SLO**, and don't let the load generator coordinate-omit the tail (§8). - **Don't tune `GOGC`/`GOMEMLIMIT` before USE confirms GC is the bottleneck** — the usual fix is fewer allocations (§3, §8). --- ## 11. Routing to Related Skills **This plugin (`go-perf-*` depth):** - `go-perf-benchmarking-statistics` — benchstat flags, p-values, geomean, sample size, perflock how-to, A/B method, CI gating commands. - `go-perf-pprof-profiling` — CPU/heap pprof subcommands, `inuse` vs `alloc`, differential profiling, labels, production profiling. - `go-perf-block-mutex-profiles` — block/mutex/goroutine profiles and the enablement gotcha. - `go-perf-execution-tracer` — `runtime/trace`, `go tool trace`, the flight recorder (1.25). - `go-perf-escape-analysis` — reading `-gcflags=-m`; what escapes to the heap. - `go-perf-gc-tuning` — `GOGC`/`GOMEMLIMIT`, `gctrace`, allocation rate as the primary lever. - `go-perf-tail-latency` — p99/p999, coordinated omission, latency histograms. - `go-perf-os-tooling` — Linux `perf`, flame graphs, perflock as a benchmark stabilizer. **Parent / sibling marketplace (`go-*` correctness):** - `go-performance` — the measure-first **overview** and the 3-step loop this skill deepens. - `go-testing-advanced` — benchmark **mechanics**: `for b.Loop()`, `b.ReportAllocs`, `b.ResetTimer`, `b.RunParallel`, the sink/DCE trap. - `go-idiomatic-discipline` — the policy root; unmeasured micro-opt vs clarity is its axis-2 ("clear AND correct") failure. - `go-version-feature-map` — version floors: container-aware `GOMAXPROCS` 1.25, Green Tea GC default 1.26. --- ## 12. Reference Files Methodology anti-patterns in LLM-generated Go performance work, each with wrong/right contrast and citations: [references/common-mistakes.md](references/common-mistakes.md) Source provenance for every claim in this skill: [references/sources.yaml](references/sources.yaml)