Go Perf Pprof Profiling

Go pprof Profiling

"When CPU profiling is enabled, the Go program stops about 100 times per second and records a sample consisting of the program counters on the currently executing goroutine's stack." — Profiling Go Programs

The headline pprof intro — -cpuprofile/-memprofile, top/list/web — belongs to go-performance §3; read it first. This skill is its deep peer: the full collection-and-reading workflow, every go tool pprof subcommand, the inuse-vs-alloc heap split, differential profiling, labels, and safe production exposure. Which diagnostic to reach for and how to read a sample statistically is go-perf-methodology. Contention (block/mutex/goroutine) → go-perf-block-mutex-profiles; off-CPU latency/scheduling → go-perf-execution-tracer; "does it escape" → go-perf-escape-analysis.

All Go below builds clean under gofmt, go vet, and go test on Go 1.26.

1. Collecting Profiles — From a Benchmark and From a Server

Two entry points cover almost everything. From a benchmark (no code changes, the fastest loop):

go test -run='^$' -bench=BenchmarkEncode -cpuprofile=cpu.prof -memprofile=mem.prof
go tool pprof cpu.prof                 # interactive REPL
go tool pprof -http=:8080 cpu.prof     # browser UI: flame graph, source, graph

From a live process, import net/http/pprof for its side effect — "The package is typically only imported for the side effect of registering its HTTP handlers" on paths that "all begin with /debug/pprof/" (net/http/pprof):

import _ "net/http/pprof" // registers /debug/pprof/ on http.DefaultServeMux

func main() {
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()
	// ... real work ...
}

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30  # 30s CPU
go tool pprof http://localhost:6060/debug/pprof/heap                # live heap

seconds=N means two different things: for the profile (CPU) and trace endpoints it sets the sample duration; for allocs/block/goroutine/heap/mutex/threadcreate it instead returns a delta profile over that window (net/http/pprof) — a built-in differential (§4). Collect one profile at a time: "precise memory profiling skews CPU profiles and goroutine blocking profiling affects scheduler trace … it is recommended to collect only a single profile at a time" (Diagnostics).

For a profile without a server, write it by hand — and runtime.GC() first so live-object stats are current:

runtime.GC() // get up-to-date statistics
if err := pprof.WriteHeapProfile(f); err != nil { // == Lookup("heap").WriteTo(f, 0)
	log.Fatal(err)
}

2. Reading a CPU Profile — Flat vs Cumulative Is the Whole Game

A profile is a sample, not ground truth: at ~100 Hz a function with <10 ms total may not appear, and "each stack sample only includes the bottom 100 stack frames," so deep recursion is under-attributed (pprof). Profile long enough to accumulate samples (the statistical reading is go-perf-methodology §2).

The subcommands, in the order you actually use them:

Flat vs cumulative is the single most-misread thing in pprof. The flat columns are "the number of samples in which the function was running (as opposed to waiting for a called function to return)"; the cumulative columns are "the number of samples in which the function appeared (either running or waiting for a called function to return)" (pprof). In the canonical example main.FindLoops "was running in 10.6% of the samples, but it was on the call stack … in 84.1% of the samples" (pprof).

• High flat, low cum → a true leaf hotspot. Optimize this function's body.
• Low flat, high cum → an expensive caller; the cost is in what it calls. list/peek to find which callee, or do less of the calling.

Default granularity is -functions; switch with -lines/-files/-addresses to localize within a big function (pprof README). Narrow with -focus=regex/-ignore=regex.

3. Heap Profiles — inuse (Leak) vs alloc (Churn)

The heap profile carries four views, and picking the wrong one is the most common heap-profiling error. "The heap profile tracks both the allocation sites for all live objects in the application memory and for all objects allocated since the program start. Pprof's -inuse_space, -inuse_objects, -alloc_space, and -alloc_objects flags select which to display, defaulting to -inuse_space (live objects, scaled by size)" (runtime/pprof).

go tool pprof -inuse_space mem.prof    # what is resident RIGHT NOW (default)
go tool pprof -alloc_space mem.prof    # what has churned through the GC
go tool pprof -alloc_objects mem.prof  # which sites allocate the most often

The split tracks the two questions cleanly:

• "Memory keeps climbing / OOM" is a live-set question → inuse_space. The profile "reports statistics as of the most recently completed garbage collection; it elides more recent allocation to avoid skewing the profile away from live data and toward garbage" (runtime/pprof) — so it shows what survived, exactly what a leak is made of.
• "GC is burning CPU / allocs/op is high" is a churn question → alloc_space/alloc_objects, "the total number of bytes allocated since the program began (including garbage-collected bytes)" (runtime/pprof). A buffer allocated and freed a million times is invisible to inuse_* but dominates alloc_*. This is the view that pairs with escape analysis — confirm the site, then go-perf-escape-analysis explains why it heap-allocates.

The memory profiler samples "approximately one block per half megabyte allocated (the '1-in-524288 sampling rate'), so these are approximations" (pprof) — read magnitudes and rankings, not exact byte counts. go test -memprofile and pprof.Lookup("allocs") default to alloc_space; Lookup("heap")/the /heap endpoint default to inuse_space (runtime/pprof).

4. Differential Profiling — Comparing Two Profiles

To see what a change (or a leak over time) actually moved, subtract one profile from another "provided the profiles are of compatible types (i.e. two heap profiles)" (pprof README). Two flags, and they are not interchangeable:

go tool pprof -diff_base=old.prof new.prof   # what got worse/better (percent vs base)
go tool pprof -base=old.prof new.prof        # leak: new live heap minus old live heap

• -diff_base is "useful for comparing two profiles. Percentages in the output are relative to the total of samples in the diff base profile" (pprof README). Use it for an A/B: hot path before vs after a change. Entries can go negative (work that got cheaper).
• -base is "useful for subtracting a cumulative profile … from another cumulative profile collected from the same program at a later time"; percentages are "relative to the difference between the total for the source profile and the total for the base profile" and all values stay positive (pprof README). This is the leak-hunt idiom: snapshot inuse_space, wait, snapshot again, -base the first from the second — what remains is what grew.

-normalize "scales the source profile so that the total of samples … is equal to the total of samples in the base profile" before subtracting — use it when the two runs had different total durations/load so percentages, not absolute counts, are comparable (pprof README). (The /heap?seconds=N delta endpoint in §1 is a server-side -base over a window.)

5. Profile Labels — Slice the Profile by Request Type

A flat CPU profile blends every code path together; labels let you ask "where does the /checkout handler spend CPU, separately from /search?" Wrap the work in pprof.Do, which "calls f with a copy of the parent context with the given labels added … Goroutines spawned while executing f will inherit the augmented label-set" (runtime/pprof):

func handler(w http.ResponseWriter, r *http.Request) {
	labels := pprof.Labels("handler", r.URL.Path, "method", r.Method)
	pprof.Do(r.Context(), labels, func(ctx context.Context) {
		serve(ctx, w, r) // CPU here is tagged; child goroutines inherit the tags
	})
}

pprof.Labels "takes an even number of strings representing key-value pairs"; crucially "only the CPU and goroutine profiles utilize any labels information" (runtime/pprof) — labels do not flow into heap/alloc profiles. Then slice in the tool with -tagfocus/-tagignore, "the most used option for selecting data in a profile based on tag values," e.g. -tagfocus=handler=/checkout, with range syntax like -tagfocus=128kb:512kb for numeric tags (pprof README). Inheritance is the payoff: tag once at the request boundary and every downstream goroutine's CPU is attributed to that request. (pprof.Do wraps the lower-level SetGoroutineLabels/WithLabels; prefer Do (runtime/pprof).)

6. Production & Continuous Profiling — Safely

In-process profiling is cheap enough to run live: "It is safe to profile programs in production, but enabling some profiles (e.g. the CPU profile) adds cost. You should expect to see performance downgrade" (Diagnostics). Two hard rules:

1. Never expose /debug/pprof/ on a public listener. import _ "net/http/pprof" registers on http.DefaultServeMux; if your app serves business traffic on that same mux, the endpoints (and the stack/heap detail they leak, plus a trivial CPU-burn DoS via ?seconds=) are internet-reachable. Bind a separate admin listener on localhost/a private port (the localhost:6060 goroutine above), or "If you are not using DefaultServeMux, you will have to register handlers with the mux you are using" (net/http/pprof) — put them behind auth on an internal-only mux.
2. One profile at a time (§1) — concurrent CPU + precise-heap collection skews both (Diagnostics).

Continuous profiling turns spot-checks into a time series. A collector scrapes the same /debug/pprof/ endpoints on a low duty cycle and stores profiles for diffing across deploys. Grafana Pyroscope is "an open source continuous profiling database" that "collects CPU and memory profiles from applications that expose pprof endpoints" and keeps "durable, long-term storage … to help you identify changes and trends over time" (Pyroscope); Parca and Polar Signals consume the same pprof format. The win is exactly §4 over time: diff today's hot path against last week's, or a regressed pod against a healthy one, instead of trying to reproduce a production hotspot on your laptop.

7. Don't

• Don't read flat time when the cost is cumulative (or vice versa). A low-flat/high-cum function isn't the hotspot — its callee is; use top -cum/peek (pprof).
• Don't debug a leak with alloc_space — churn ≠ live set. A leak lives in inuse_space; churn lives in alloc_* (runtime/pprof).
• Don't trust a short profile. At ~100 Hz a 50 ms benchmark yields ~5 samples — noise; profile longer (pprof).
• Don't collect CPU and precise-memory profiles at once — they skew each other; one at a time (Diagnostics).
• Don't expose net/http/pprof on your public mux — bind a private admin listener or gate it behind auth (net/http/pprof).
• Don't expect labels in a heap profile — only CPU and goroutine profiles carry them (runtime/pprof).
• Don't confuse -diff_base with -base — -diff_base for an A/B (percent vs base, negatives allowed), -base for subtracting cumulative snapshots (positives, leak hunt) (pprof README).
• Don't forget runtime.GC() before a manual heap profile — without it inuse_* includes not-yet-collected garbage (runtime/pprof).
• Don't reach for pprof to explain latency or blocking — a CPU profile is blind to off-CPU waits; that's the tracer and block/mutex profiles.

8. Routing to Related Skills

This plugin (go-perf-* depth):

• go-perf-methodology — which diagnostic answers which question; reading a profile as a statistical sample; the order of operations.
• go-perf-block-mutex-profiles — block/mutex/goroutine profiles and the SetBlockProfileRate/SetMutexProfileFraction enablement gotcha (off-CPU contention).
• go-perf-execution-tracer — runtime/trace + go tool trace for latency/scheduling/GC the CPU profile can't see; the flight recorder.
• go-perf-escape-analysis — once alloc_* names a site, -gcflags=-m says why it heap-allocates.
• go-perf-gc-tuning — when the heap profile says GC pressure is the cost.
• go-perf-pgo — the CPU profile you collected here is the PGO input (default.pgo).
• go-perf-os-tooling — Linux perf/flame graphs when the cost is in the kernel pprof can't attribute.

Parent / sibling marketplace (go-* correctness):

• go-performance — the altitude pprof intro (top/list/web) this skill deepens.
• go-testing-advanced — benchmark mechanics that feed -cpuprofile/-memprofile.

9. Reference Files

pprof anti-patterns in LLM-generated Go performance work, each with wrong/right contrast and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml