Go Perf Benchmarking Statistics

Go Benchmarking Statistics

"Each benchmark should be run at least 10 times to gather a statistically significant sample of results." — benchstat

The benchmark loop — for b.Loop(), the sink/dead-code trap, b.ReportAllocs, b.ResetTimer, b.RunParallel — belongs to go-testing-advanced; write the benchmark there first. The concept of why a single run lies (and the USE method behind it) belongs to go-perf-methodology. This skill owns the layer in between: the commands and statistics that turn benchmark output into a trustworthy A/B — benchstat, -count, the p-value/geomean/~, the right unit, a reproducible machine, and the CI gate.

All Go below builds clean under gofmt, go vet, and go test on Go 1.26.

1. The A/B Workflow — Collect Two Files, Diff Them

benchstat "computes statistical summaries and A/B comparisons of Go benchmarks" and reads "the Go benchmark format … such as the output of go test -bench ." (benchstat). The canonical A/B is two files from the same benchmark, before and after the change:

git stash                                                   # or: git switch main
go test -run='^$' -bench=. -benchmem -count=10 > old.txt
git stash pop                                               # or: git switch feature
go test -run='^$' -bench=. -benchmem -count=10 > new.txt
benchstat old.txt new.txt

Two flags carry this and are constant LLM omissions:

• -run='^$' disables unit tests during the bench so their work and output don't pollute the run — without it the default -run=. runs the package's tests too (Dave Cheney, How to Write Benchmarks).
• -benchmem adds the B/op and allocs/op columns. allocs/op is deterministic per code path, so it is the cheapest, least-noisy signal you have — never leave it off.

A bare benchstat new.txt (one file) is a summary, not a comparison — useful to read the variance, but it answers "how stable is this?", not "did it change?".

2. Reading the Output — ±, p, ~, n, geomean

                  │   old.txt    │              new.txt               │
                  │   sec/op     │   sec/op     vs base               │
Encode/json-48      1.718µ ± 1%    1.423µ ± 1%  -17.20% (p=0.000 n=10)
Encode/gob-48       3.066µ ± 0%    3.070µ ± 2%         ~ (p=0.446 n=10)
geomean             2.295µ         2.090µ       -8.94%

Every field is load-bearing:

• ± 1% is the spread, not a guess: "benchstat computes the median and the 95% confidence interval for the median" (benchstat). A ± above ~3–5% means a noisy machine (§6) — fix the machine before trusting the delta. The center value is the median, not the mean, because benchmark noise is one-sided (interference only ever makes a run slower).
• p=0.000 is significance: "the p-value measures how likely it is that any differences were due to random chance (i.e., noise)." By default benchstat "uses non-parametric statistics: median for summaries, and the Mann–Whitney U-test for A/B comparisons" (benchstat).
• ~ is a result: it means benchstat "did not detect a statistically significant difference" — for gob above, "a p-value of 0.446, meaning it's very likely the differences … are simply due to random chance" (benchstat). Read ~ as "the change bought nothing measurable — don't ship the added complexity."
• n=10 is the surviving sample size after benchstat discards outliers; if it drops well below your -count, the run was unstable.
• geomean is the suite verdict: the last row "shows the geometric mean of each column, giving an overall picture of how the benchmarks changed … if sec/op for one of them increases by a factor of 2, then the sec/op geomean will increase by a factor of ⁿ√2" (benchstat). It is proportional, so it can't be skewed by one huge absolute number.

The α threshold is 0.05: benchstat "is expected to show a difference 5% of the time even if there is no difference" (benchstat). In a 40-row table ~2 "significant" rows are pure chance — judge the geomean and coherent clusters, not a lone flipped row.

3. Sample Size — -count Drives the Whole Thing

The U-test has no power without a sample: "each benchmark should be run at least 10 times" (benchstat). -count=N runs each benchmark N times within one binary invocation; that N is the n the test gets. -count=1 (the default) gives benchstat one data point per side and forces p=1.00 / ~ on everything — it literally cannot compute significance.

Prefer interleaving for long suites: noise drifts over minutes (thermals, background jobs), so running all of old then all of new can confound the drift with the change. Tools like benchstat's companion benchcmp workflows and shell loops alternate the two binaries; at minimum, run on a machine pinned per §6 so drift is bounded. For a borderline result, raising -count (20, 30) tightens the ± and resolves a near-threshold p-value rather than guessing.

4. Grouping a Matrix — -col, -row, -table, -filter

When a benchmark varies along axes (b.Run("format=json", …), multiple GOMAXPROCS, several input sizes), benchstat pivots it. By default it "groups results into tables using all file-level configuration keys, then within each table … into rows by .fullname … and compares across columns by .file" (benchstat) — i.e. the implicit form is -table .config -row .fullname -col .file. Override the axes:

benchstat -col /format old.txt new.txt        # compare json vs gob vs proto side by side
benchstat -row .fullname -col .file a.txt b.txt c.txt   # three files across columns
benchstat -filter "/format:json .unit:(ns/op OR B/op)" old.txt new.txt
benchstat -ignore goos old.txt new.txt        # collapse a key out of the grouping

The sub-name keys come from b.Run labels written key=value (so name subtests format=json, not json); .unit filters which metric columns appear; -filter takes a boolean expression over keys (benchstat). This is how a single -count=10 run of a 3×3 matrix becomes one readable table instead of nine eyeballed numbers.

5. Choosing the Right Unit — MB/s and Custom Metrics

ns/op answers "how long," not "how much work per second." For throughput, set the byte count and the framework derives the rate: SetBytes "records the number of bytes processed in a single operation. If this is called, the benchmark will report ns/op and MB/s" (testing).

func BenchmarkDecode(b *testing.B) {
	payload := loadPayload() // N bytes
	b.SetBytes(int64(len(payload)))
	b.ReportAllocs()
	for b.Loop() {
		_, _ = Decode(payload)
	}
}

MB/s is the right axis for codecs, compressors, and I/O: a change that adds 5% ns/op but processes a larger batch can still be a throughput win, and only MB/s shows it. For any other unit — items, requests, cache hits — use ReportMetric: it "adds 'n unit' to the reported benchmark results. If the metric is per-iteration, the caller should divide by b.N, and by convention units should end in '/op'" (testing). benchstat reports any column you emit, so a custom rows/op flows straight into the A/B table.

b.ReportMetric(float64(rowsProcessed)/float64(b.N), "rows/op")

ReportMetric "overrides any previously reported value for the same unit," can override built-ins like allocs/op, and "panics if unit … contains any whitespace"; "setting 'ns/op' to 0 will suppress that built-in metric" (testing). (With b.Loop, b.N is still the iteration count to divide by.)

6. A Reproducible Machine — Or the Statistics Are Noise

§2's ± and p are only meaningful if the machine holds still between the two runs. The dominant culprit is CPU frequency scaling: turbo boost and thermal throttling make the same binary benchmark 20%+ differently run-to-run, burying any real delta inside the noise floor.

perflock fixes both frequency and contention. It is "a simple locking wrapper for running benchmarks on shared hosts" that acquires "a system-wide lock while running command" (perflock). Two defaults matter:

• Exclusive mode is the default — it "prevents any other perflock'd command from running," which serializes benchmarks so two never fight for CPU and cache at once (perflock).
• -governor defaults to 90% — it sets the CPU frequency "between the min and max while running command," pinning the clock below turbo so it stays constant, and restores the governor on release (perflock).

# install once (Linux): build, then the daemon holds the lock
go build ./cmd/perflock && sudo ./install.bash      # or: sudo -b perflock -daemon
perflock -governor 90% go test -run='^$' -bench=. -benchmem -count=10 > new.txt

Beyond perflock, on a bench host: pin the governor to performance and disable turbo if you do not use perflock; taskset/isolcpus to keep the OS scheduler off the bench cores; set GOMAXPROCS explicitly so the parallelism is fixed; close browsers and background jobs (a quiet machine, ideally bare metal over a noisy-neighbor VM). Note Go 1.25 makes the default GOMAXPROCS container-aware — under a cgroup CPU limit the default drops to that limit, so verify it when benching in a container (Go 1.25). Without governor pinning + serialization + -count≥10, the delta is inside the noise and the U-test has nothing to bite on.

7. Gating Regressions in CI

A regression that isn't gated lands silently. Run the suite on the PR and its base on the same runner, diff with benchstat, and fail the build on a significant regression:

git switch --detach origin/main
go test -run='^$' -bench=. -benchmem -count=10 > base.txt
git switch --detach "$PR_SHA"
go test -run='^$' -bench=. -benchmem -count=10 > pr.txt
benchstat base.txt pr.txt        # fail if a tracked bench regresses past threshold

Principles that make the gate survive a shared, noisy CI runner:

• Gate allocs/op, not ns/op first. Alloc counts are deterministic per path, so a 1→3 jump is a real regression even when timing is noisy — the most reliable cheap gate. Time thresholds on a shared runner are fragile.
• Compare relative A/B, never an absolute ns/op floor. The runner's clock varies; PR-vs-base in the same job cancels the runner. Treat ~ as "no regression."
• Budgets are goals made executable ("0 allocs/op on the hot encode path"; "decode ≥ 500 MB/s"). go-perfbook's step one — set goals and confirm you're not meeting them — turns "fast enough?" into a CI assertion (go-perfbook).

8. Don't

• Don't declare a winner from two single ns/op numbers. -count=1 gives the U-test no power; use -count≥10 + benchstat and accept ~ as "no win" (benchstat).
• Don't omit -benchmem. allocs/op is the cheapest, most deterministic signal — and the best CI gate.
• Don't leave -run=. while benchmarking — the package's tests run too and pollute the numbers (Dave Cheney).
• Don't trust a delta when ± is large. A wide confidence interval means a noisy machine (§6), not a real change — pin the clock and re-run.
• Don't chase a lone significant row in a big table; at α=0.05 ~1 in 20 is chance — read the geomean (benchstat).
• Don't benchmark on an unpinned laptop. Turbo and noisy neighbors vary the same code 20%+; serialize and pin with perflock (perflock).
• Don't report ns/op for a throughput question — set b.SetBytes for MB/s or b.ReportMetric for a custom unit (testing).
• Don't gate CI on an absolute ns/op floor — compare PR vs base in the same job; gate allocs/op first.

9. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-methodology — why one run lies (the statistical concept), the USE method, micro-vs-macro design, the order of operations. This skill is its command-level executor.
• go-perf-pprof-profiling — once benchstat says something changed, where the time/allocs go (CPU/heap pprof, -diff_base).
• go-perf-os-tooling — perflock as one piece of a stabilized bench host alongside Linux perf and flame graphs.

Parent / sibling marketplace (go-* correctness):

• go-testing-advanced — benchmark mechanics: for b.Loop(), the sink/DCE trap, b.ReportAllocs, b.ResetTimer, b.RunParallel. Write the benchmark there; measure it here.
• go-performance — the altitude benchstat mention and the measure-first loop this deepens.
• go-version-feature-map — version floors: b.Loop 1.24, container-aware GOMAXPROCS 1.25.

10. Reference Files

Benchmarking/statistics anti-patterns in LLM-generated Go, each with wrong/right contrast and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml