ctoth Go Profile-Guided Optimization (PGO)
The headline — "feed a real CPU profile back into the compiler; commit default.pgo" — belongs to go-performance §6; read it first. This skill is its deep peer for PGO: where the profile must come from and why, the default.pgo lifecycle, what PGO actually does to hot code, the honest payoff, and the iteration/merge mechanics. The two optimizations PGO unlocks route out — inlining depth to go-perf-inlining, devirtualization to go-perf-compiler-intrinsics — because PGO does not invent them, it just applies them more aggressively where a profile proves a call site is hot.
All Go below builds clean under gofmt, go vet, and go test on Go 1.26.
1. What PGO Is, and the Version Floor
"Starting in Go 1.20, the Go compiler supports profile-guided optimization (PGO)" (PGO guide), but 1.20 was a preview. PGO is GA from Go 1.21: "PGO support in Go 1.21 is ready for general production use!" (PGO blog). In the 1.21 release notes, "Profile-guide optimization (PGO) … is now ready for general use," and "The -pgo build flag now defaults to -pgo=auto" (Go 1.21).
PGO "enables additional optimizations on code identified as hot by profiles of production workloads" (Go 1.21). It is not a new optimization — it is evidence that tells the compiler which call sites are worth spending its existing budgets on. Gate any PGO idiom on the module's go directive being ≥ 1.21 (see go-version-feature-map).
2. default.pgo — The One Convention That Matters
"The standard approach … is to store a pprof CPU profile with filename default.pgo in the main package directory of the profiled binary. By default, go build will detect default.pgo files automatically and enable PGO" (PGO guide). "The Go toolchain will automatically enable PGO when it finds a profile named default.pgo in the main package directory" (PGO blog).
myapp/
main.go
default.pgo <- CPU profile, here, named exactly this
go build ./... # PGO auto-enabled; no -pgo flag, no code change
Three consequences:
- No flag, no opt-in. The file's name and location are the whole interface. A profile named anything else, or in any other directory, is silently ignored.
- It must be a CPU profile. A heap/block/mutex profile is not valid PGO input — PGO consumes the same pprof CPU profile pprof's
profileendpoint produces (§4). - Override explicitly when needed:
go build -pgo=offdisables it;go build -pgo=/path/to/foo.pprofpoints at a specific file. The default is-pgo=auto(Go 1.21).
3. What PGO Actually Buys — Two Optimizations, Both Routed
PGO is not magic; it sharpens two existing optimizations using the profile's hot-site data.
Inlining — more aggressive on proven-hot calls. "The compiler may decide to more aggressively inline functions which the profile indicates are called frequently" (PGO guide). Without a profile the compiler inlines by a static cost budget (~80 nodes); with one, it spends extra budget on the call sites the profile says dominate. The mechanics of that budget, what blocks inlining, and reading -gcflags=-m → go-perf-inlining.
Devirtualization — hot interface calls become direct calls. "PGO builds can now devirtualize some interface method calls, adding a concrete call to the most common callee. This enables further optimization, such as inlining the callee" (Go 1.21). An interface call is an indirect jump the compiler normally cannot inline through; when the profile shows one concrete type dominates a call site, PGO emits a direct call to it (guarded, falling back to the interface call for other types) — and a direct call can then be inlined. The call-overhead/dispatch model → go-perf-compiler-intrinsics.
The takeaway for this skill: you do not write any code to get these. You supply a representative profile, and the compiler routes its budget to where the profile says it pays.
4. Collecting the Profile — From Production, Not a Bench
This is the single decision that makes or breaks PGO. "For best results, it is important that profiles are representative of actual behavior in the application's production environment. Using an unrepresentative profile is likely to result in a binary with little to no improvement in production. … collecting profiles directly from the production environment is recommended, and is the primary method that Go's PGO is designed for" (PGO guide).
Why production, not a microbenchmark: PGO only optimizes the call sites the profile flags as hot. A microbenchmark exercises one tiny path under cache-hot, GC-quiet, single-input conditions (see go-perf-methodology §5) — so it flags the wrong sites hot, and the compiler spends its extra budget on code production barely runs. A profile from real traffic flags the sites that actually dominate.
Profiles "from runtime/pprof and net/http/pprof can be used directly as the compiler input" (PGO guide) — the CPU profile you already collect is the PGO input. The how of collecting safely (endpoint exposure, sampling window, labels) lives in go-perf-pprof-profiling; the brief version:
import _ "net/http/pprof" // registers /debug/pprof/ on the default mux
// long-running server: go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }()
# pull ~30s of CPU profile from a representative production instance
curl -o cpu.pprof "http://localhost:6060/debug/pprof/profile?seconds=30"
cp cpu.pprof myapp/default.pgo
Or in-process with runtime/pprof.StartCPUProfile(w) / StopCPUProfile() (runtime/pprof). Whether the profile is representative enough to act on is a methodology judgment → go-perf-methodology.
The whole first-time setup, end to end:
# 1. capture a representative production CPU profile (~30s of real traffic)
curl -o cpu.pprof "http://prod-instance:6060/debug/pprof/profile?seconds=30"
# 2. drop it in the main package under the exact name
cp cpu.pprof ./cmd/server/default.pgo
# 3. commit it as a versioned build input
git add ./cmd/server/default.pgo && git commit -m "Add CPU profile for PGO"
# 4. build — PGO is auto-enabled, no flag, no code change
go build ./cmd/server
From here it is maintenance: re-pull and re-commit periodically (§6).
5. Commit default.pgo — Reproducible Builds
The profile is a build input, so it lives in source control next to the code it optimizes. "Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds" (PGO guide). "Storing profiles alongside your source code ensures that users automatically have access to the profile simply by fetching the repository … and that builds remain reproducible" (PGO blog).
Concretely: a default.pgo in .gitignore, or generated only in CI, means two checkouts of the same commit produce different binaries — a reproducibility break, and contributors who go get your module silently lose the optimization. Commit the file. It is typically a few hundred KB; treat it like any other versioned build input.
6. Iteration, Staleness, and Convergence
Refresh as code drifts. A profile captures today's hot paths; refactors move them. "Degradation will slowly accumulate over time since code is rarely refactored back to its old form, so it is important to collect new profiles regularly to limit source skew from production" (PGO guide). Wire profile refresh into a periodic job, not a one-time setup.
The chicken-and-egg worry is unfounded. You might fear a feedback loop: you build with PGO, profile that binary, rebuild with the new profile, and oscillate. Go is designed against this. "Go PGO is generally robust to skew between the profiled version of an application and the version building with the profile, as well as to building with profiles collected from already-optimized binaries" (PGO guide). The property is named iterative stability — "the prevention of cycles of variable performance in successive PGO builds (e.g., build #1 is fast, build #2 is slow, build #3 is fast, etc)" (PGO guide). So profiling a PGO-built production binary and feeding that back is fine — builds converge, they do not oscillate.
Merging multiple profiles. Production is many instances and workload shapes; combine their profiles rather than betting on one. "The pprof tool can merge multiple profiles" (PGO guide):
go tool pprof -proto a.pprof b.pprof c.pprof > default.pgo
This produces one representative default.pgo from several captures — useful when one instance over-represents a single endpoint.
7. The Honest Payoff
PGO is a free, modest win — not an algorithmic one. "As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%" (PGO guide); the 1.21 announcement measured "between 2% and 7% CPU usage improvements from enabling PGO" (PGO blog), typically a few percent. Frame expectations accordingly:
- It is worth doing because it costs almost nothing (one committed file, no code change) and compounds with every build — not because it is a large lever. Algorithmic and allocation wins dwarf it (
go-perf-methodology§6). - The gain only lands on hot code the profile captured. A non-representative profile "is likely to result in a binary with little to no improvement" (PGO guide) — the most common reason "PGO did nothing for me."
- Verify the win the same way as any optimization: benchstat the PGO build against the non-PGO build, n≥10 (→
go-perf-benchmarking-statistics). A~means this binary didn't benefit measurably; don't claim a number you didn't measure.
8. Scope & Cost Caveats — from the FAQ
A few facts that change how you reason about PGO at the program level:
- PGO optimizes the whole program, not just your code. "PGO in Go applies to the entire program. All packages are rebuilt to consider potential profile-guided optimizations, including standard library packages" and "including packages in dependencies. This means that the unique way your application uses a dependency impacts the optimizations applied to that dependency" (PGO guide). A hot path through a JSON or compression library gets devirtualized/inlined for your usage pattern.
- Profiles are portable across platforms. "The format of the profiles is equivalent across OS and architecture configurations, so they may be used across different configurations. For example, a profile collected from a
linux/arm64binary may be used in awindows/amd64build" (PGO guide). Onedefault.pgoserves every cross-compile target. - One binary, many workloads = no perfect profile. "A single binary used for different types of workloads (e.g., a database used in a read-heavy way in one service, and write-heavy in another service) may have different hot components, which benefit from different optimizations" (PGO guide). Merging (§6) blends them; per-deployment profiles serve each better.
- It costs build time and a little binary size. "Enabling PGO builds will likely cause measurable increases in package build times," and "PGO can result in slightly larger binaries due to additional function inlining" (PGO guide). Both are usually a fair trade for a free runtime win, but budget for them in CI.
9. Don't
- Don't profile a microbenchmark for PGO. It flags the wrong sites hot; collect from production (PGO guide).
- Don't
.gitignoredefault.pgo. It's a build input; omitting it breaks reproducibility and deniesgo getusers the profile (PGO blog). - Don't feed a heap/block/mutex profile to PGO — it consumes a CPU profile only (§2, §4).
- Don't name the file anything but
default.pgo, or place it outside the main package — auto-detection keys on exactly that (PGO guide). - Don't fear iterating on a PGO-built binary's profile — iterative stability means builds converge, not oscillate (PGO guide).
- Don't set the profile once and forget it — refresh as code drifts or the win erodes (PGO guide).
- Don't expect more than a few percent, and don't claim even that without a benchstat A/B (it's a measured 2-14%) (PGO guide).
- Don't reach for PGO before the cheap wins. It's the last few percent, after algorithm, allocations, and the free runtime wins (
go-performance§6–7).
10. Routing to Related Skills
This plugin (go-perf-* depth):
go-perf-inlining— the inlining budget, mid-stack inlining, what blocks it, reading-gcflags=-m; PGO makes this more aggressive on hot sites.go-perf-compiler-intrinsics— interface call dispatch and the call-overhead model behind PGO devirtualization.go-perf-pprof-profiling— collecting the CPU profile safely (net/http/pprof exposure, sampling, labels) that becomesdefault.pgo.go-perf-methodology— whether a profile is representative enough to act on, and where PGO sits in the order of operations (last, modest).go-perf-benchmarking-statistics— proving the PGO build is actually faster with benchstat, n≥10.
Parent / sibling marketplace (go-* correctness):
go-performance— the measure-first overview and the headline PGO paragraph (§6) this skill deepens.go-version-feature-map— version floors: PGO GA 1.21,-pgo=autodefault.
11. Reference Files
PGO anti-patterns in LLM-generated Go, each with wrong/right contrast and citations:
references/common-mistakes.md
Source provenance for every claim in this skill:
references/sources.yaml