# Go Perf Benchmarking Statistics > Guides trustworthy Go benchmark measurement at the command level — running each side with -count≥10 into old.txt/new.txt and feeding both to benchstat, reading its ± confidence interval, p-value (Mann–Whitney U), geomean, and the "~" no-significant-difference result, choosing the right unit (b.SetBytes for MB/s, b.ReportMetric for custom throughput), grouping a matrix with -col/-row/-filter, building a reproducible environment (perflock's 90% governor + serialization, frequency scaling. Auto-invokes when comparing benchmark runs, running benchstat, writing a perf A/B, or stabilizing a bench machine, and on "is this benchmark result real", "compare these benchmarks", "did this actually get faster", "my benchmark is noisy", "set up a perf regression test". Routes benchmark loop mechanics (for b.Loop, the sink/DCE trap, ResetTimer, RunParallel) to go-testing-advanced and the why-one-run-lies concept to go-perf-methodology. - Skill: `ctoth/go-perf-benchmarking-statistics` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add ctoth/go-perf-benchmarking-statistics` - Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-benchmarking-statistics/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: ctoth (https://skillmd.com/u/ctoth) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ctoth/go-perf-benchmarking-statistics --- # Go Benchmarking Statistics > "Each benchmark should be run at least 10 times to gather a statistically significant sample of results." > — [benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) The benchmark *loop* — `for b.Loop()`, the sink/dead-code trap, `b.ReportAllocs`, `b.ResetTimer`, `b.RunParallel` — belongs to **`go-testing-advanced`**; write the benchmark there first. The *concept* of why a single run lies (and the USE method behind it) belongs to **`go-perf-methodology`**. This skill owns the layer in between: the **commands and statistics** that turn benchmark output into a trustworthy A/B — `benchstat`, `-count`, the p-value/geomean/`~`, the right *unit*, a reproducible *machine*, and the CI gate. All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.26. --- ## 1. The A/B Workflow — Collect Two Files, Diff Them benchstat "computes statistical summaries and A/B comparisons of Go benchmarks" and reads "the Go benchmark format … such as the output of `go test -bench .`" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). The canonical A/B is two files from the *same* benchmark, before and after the change: ```sh git stash # or: git switch main go test -run='^$' -bench=. -benchmem -count=10 > old.txt git stash pop # or: git switch feature go test -run='^$' -bench=. -benchmem -count=10 > new.txt benchstat old.txt new.txt ``` Two flags carry this and are constant LLM omissions: - **`-run='^$'`** disables unit tests during the bench so their work and output don't pollute the run — without it the default `-run=.` runs the package's tests too ([Dave Cheney, How to Write Benchmarks](https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go)). - **`-benchmem`** adds the `B/op` and `allocs/op` columns. `allocs/op` is *deterministic* per code path, so it is the cheapest, least-noisy signal you have — never leave it off. A bare `benchstat new.txt` (one file) is a *summary*, not a comparison — useful to read the variance, but it answers "how stable is this?", not "did it change?". --- ## 2. Reading the Output — `±`, `p`, `~`, `n`, geomean ``` │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ Encode/json-48 1.718µ ± 1% 1.423µ ± 1% -17.20% (p=0.000 n=10) Encode/gob-48 3.066µ ± 0% 3.070µ ± 2% ~ (p=0.446 n=10) geomean 2.295µ 2.090µ -8.94% ``` Every field is load-bearing: - **`± 1%`** is the spread, not a guess: "benchstat computes the median and the 95% confidence interval for the median" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). A `±` above ~3–5% means a noisy machine (§6) — *fix the machine before trusting the delta.* The center value is the **median**, not the mean, because benchmark noise is one-sided (interference only ever makes a run slower). - **`p=0.000`** is significance: "the p-value measures how likely it is that any differences were due to random chance (i.e., noise)." By default benchstat "uses non-parametric statistics: median for summaries, and the Mann–Whitney U-test for A/B comparisons" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). - **`~`** is a *result*: it means benchstat "did not detect a statistically significant difference" — for `gob` above, "a p-value of 0.446, meaning it's very likely the differences … are simply due to random chance" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). Read `~` as "the change bought nothing measurable — don't ship the added complexity." - **`n=10`** is the surviving sample size after benchstat discards outliers; if it drops well below your `-count`, the run was unstable. - **geomean** is the suite verdict: the last row "shows the geometric mean of each column, giving an overall picture of how the benchmarks changed … if sec/op for one of them increases by a factor of 2, then the sec/op geomean will increase by a factor of ⁿ√2" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). It is proportional, so it can't be skewed by one huge absolute number. The α threshold is 0.05: benchstat "is *expected* to show a difference 5% of the time even if there is no difference" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). In a 40-row table ~2 "significant" rows are pure chance — judge the geomean and coherent clusters, not a lone flipped row. --- ## 3. Sample Size — `-count` Drives the Whole Thing The U-test has no power without a sample: "each benchmark should be run at least 10 times" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). `-count=N` runs each benchmark N times *within one binary invocation*; that N is the `n` the test gets. `-count=1` (the default) gives benchstat one data point per side and forces `p=1.00` / `~` on everything — it literally cannot compute significance. Prefer **interleaving** for long suites: noise drifts over minutes (thermals, background jobs), so running all of `old` then all of `new` can confound the drift with the change. Tools like `benchstat`'s companion `benchcmp` workflows and shell loops alternate the two binaries; at minimum, run on a machine pinned per §6 so drift is bounded. For a borderline result, raising `-count` (20, 30) tightens the `±` and resolves a near-threshold p-value rather than guessing. --- ## 4. Grouping a Matrix — `-col`, `-row`, `-table`, `-filter` When a benchmark varies along axes (`b.Run("format=json", …)`, multiple GOMAXPROCS, several input sizes), benchstat pivots it. By default it "groups results into tables using all file-level configuration keys, then within each table … into rows by `.fullname` … and compares across columns by `.file`" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)) — i.e. the implicit form is `-table .config -row .fullname -col .file`. Override the axes: ```sh benchstat -col /format old.txt new.txt # compare json vs gob vs proto side by side benchstat -row .fullname -col .file a.txt b.txt c.txt # three files across columns benchstat -filter "/format:json .unit:(ns/op OR B/op)" old.txt new.txt benchstat -ignore goos old.txt new.txt # collapse a key out of the grouping ``` The sub-name keys come from `b.Run` labels written `key=value` (so name subtests `format=json`, not `json`); `.unit` filters which metric columns appear; `-filter` takes a boolean expression over keys ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). This is how a single `-count=10` run of a 3×3 matrix becomes one readable table instead of nine eyeballed numbers. --- ## 5. Choosing the Right Unit — MB/s and Custom Metrics `ns/op` answers "how long," not "how much work per second." For throughput, set the byte count and the framework derives the rate: `SetBytes` "records the number of bytes processed in a single operation. If this is called, the benchmark will report ns/op and MB/s" ([testing](https://pkg.go.dev/testing#B.SetBytes)). ```go func BenchmarkDecode(b *testing.B) { payload := loadPayload() // N bytes b.SetBytes(int64(len(payload))) b.ReportAllocs() for b.Loop() { _, _ = Decode(payload) } } ``` `MB/s` is the right axis for codecs, compressors, and I/O: a change that adds 5% `ns/op` but processes a larger batch can still be a throughput *win*, and only MB/s shows it. For any other unit — items, requests, cache hits — use `ReportMetric`: it "adds 'n unit' to the reported benchmark results. If the metric is per-iteration, the caller should divide by `b.N`, and by convention units should end in '/op'" ([testing](https://pkg.go.dev/testing#B.ReportMetric)). benchstat reports any column you emit, so a custom `rows/op` flows straight into the A/B table. ```go b.ReportMetric(float64(rowsProcessed)/float64(b.N), "rows/op") ``` `ReportMetric` "overrides any previously reported value for the same unit," can override built-ins like `allocs/op`, and "panics if unit … contains any whitespace"; "setting 'ns/op' to 0 will suppress that built-in metric" ([testing](https://pkg.go.dev/testing#B.ReportMetric)). (With `b.Loop`, `b.N` is still the iteration count to divide by.) --- ## 6. A Reproducible Machine — Or the Statistics Are Noise §2's `±` and `p` are only meaningful if the machine holds still between the two runs. The dominant culprit is **CPU frequency scaling**: turbo boost and thermal throttling make the *same* binary benchmark 20%+ differently run-to-run, burying any real delta inside the noise floor. **perflock** fixes both frequency and contention. It is "a simple locking wrapper for running benchmarks on shared hosts" that acquires "a system-wide lock while running command" ([perflock](https://github.com/aclements/perflock)). Two defaults matter: - **Exclusive mode is the default** — it "prevents any other perflock'd command from running," which *serializes* benchmarks so two never fight for CPU and cache at once ([perflock](https://github.com/aclements/perflock)). - **`-governor` defaults to 90%** — it sets the CPU frequency "between the min and max while running command," pinning the clock *below* turbo so it stays constant, and restores the governor on release ([perflock](https://github.com/aclements/perflock)). ```sh # install once (Linux): build, then the daemon holds the lock go build ./cmd/perflock && sudo ./install.bash # or: sudo -b perflock -daemon perflock -governor 90% go test -run='^$' -bench=. -benchmem -count=10 > new.txt ``` Beyond perflock, on a bench host: pin the governor to `performance` and disable turbo if you do not use perflock; `taskset`/`isolcpus` to keep the OS scheduler off the bench cores; set `GOMAXPROCS` explicitly so the parallelism is fixed; close browsers and background jobs (a quiet machine, ideally bare metal over a noisy-neighbor VM). Note **Go 1.25 makes the default `GOMAXPROCS` container-aware** — under a cgroup CPU limit the default drops to that limit, so verify it when benching in a container ([Go 1.25](https://go.dev/doc/go1.25)). Without governor pinning + serialization + `-count≥10`, the delta is inside the noise and the U-test has nothing to bite on. --- ## 7. Gating Regressions in CI A regression that isn't gated lands silently. Run the suite on the PR *and its base on the same runner*, diff with benchstat, and fail the build on a significant regression: ```sh git switch --detach origin/main go test -run='^$' -bench=. -benchmem -count=10 > base.txt git switch --detach "$PR_SHA" go test -run='^$' -bench=. -benchmem -count=10 > pr.txt benchstat base.txt pr.txt # fail if a tracked bench regresses past threshold ``` Principles that make the gate survive a shared, noisy CI runner: - **Gate `allocs/op`, not `ns/op` first.** Alloc counts are deterministic per path, so a 1→3 jump is a real regression even when timing is noisy — the most reliable cheap gate. Time thresholds on a shared runner are fragile. - **Compare relative A/B, never an absolute `ns/op` floor.** The runner's clock varies; PR-vs-base in the same job cancels the runner. Treat `~` as "no regression." - **Budgets are goals made executable** ("0 allocs/op on the hot encode path"; "decode ≥ 500 MB/s"). go-perfbook's step one — set goals and confirm you're not meeting them — turns "fast enough?" into a CI assertion ([go-perfbook](https://github.com/dgryski/go-perfbook)). --- ## 8. Don't - **Don't declare a winner from two single `ns/op` numbers.** `-count=1` gives the U-test no power; use `-count≥10` + benchstat and accept `~` as "no win" ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). - **Don't omit `-benchmem`.** `allocs/op` is the cheapest, most deterministic signal — and the best CI gate. - **Don't leave `-run=.`** while benchmarking — the package's tests run too and pollute the numbers ([Dave Cheney](https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go)). - **Don't trust a delta when `±` is large.** A wide confidence interval means a noisy machine (§6), not a real change — pin the clock and re-run. - **Don't chase a lone significant row** in a big table; at α=0.05 ~1 in 20 is chance — read the geomean ([benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)). - **Don't benchmark on an unpinned laptop.** Turbo and noisy neighbors vary the *same* code 20%+; serialize and pin with perflock ([perflock](https://github.com/aclements/perflock)). - **Don't report `ns/op` for a throughput question** — set `b.SetBytes` for MB/s or `b.ReportMetric` for a custom unit ([testing](https://pkg.go.dev/testing#B.SetBytes)). - **Don't gate CI on an absolute `ns/op` floor** — compare PR vs base in the same job; gate `allocs/op` first. --- ## 9. Routing to Related Skills **This plugin (`go-perf-*` depth):** - `go-perf-methodology` — *why* one run lies (the statistical concept), the USE method, micro-vs-macro design, the order of operations. This skill is its command-level executor. - `go-perf-pprof-profiling` — once benchstat says *something* changed, where the time/allocs go (CPU/heap pprof, `-diff_base`). - `go-perf-os-tooling` — perflock as one piece of a stabilized bench host alongside Linux `perf` and flame graphs. **Parent / sibling marketplace (`go-*` correctness):** - `go-testing-advanced` — benchmark **mechanics**: `for b.Loop()`, the sink/DCE trap, `b.ReportAllocs`, `b.ResetTimer`, `b.RunParallel`. Write the benchmark there; measure it here. - `go-performance` — the altitude `benchstat` mention and the measure-first loop this deepens. - `go-version-feature-map` — version floors: `b.Loop` 1.24, container-aware `GOMAXPROCS` 1.25. --- ## 10. Reference Files Benchmarking/statistics anti-patterns in LLM-generated Go, each with wrong/right contrast and citations: [references/common-mistakes.md](references/common-mistakes.md) Source provenance for every claim in this skill: [references/sources.yaml](references/sources.yaml)