# Go Perf Pprof Profiling

> Guides the full pprof workflow at depth — collecting CPU and heap profiles from benchmarks and net/http/pprof, go tool pprof top/list/peek/disasm/web (flat vs -cum), the heap inuse-vs-alloc split, -diff_base/-base differential profiling, slicing by request type with pprof.Labels/pprof.Do, and safe production net/http/pprof plus continuous profiling (Pyroscope/Parca). Auto-invokes on "profile this", "what's allocating", "read this pprof", "where's the CPU going", "set up production profiling". Routes contention (block/mutex/goroutine) profiles to go-perf-block-mutex-profiles and the execution tracer to go-perf-execution-tracer.

- Skill: `ctoth/go-perf-pprof-profiling` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-pprof-profiling`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-pprof-profiling/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-pprof-profiling

---


# Go pprof Profiling

> "When CPU profiling is enabled, the Go program stops about 100 times per second and records a sample consisting of the program counters on the currently executing goroutine's stack."
> — [Profiling Go Programs](https://go.dev/blog/pprof)

The headline pprof intro — `-cpuprofile`/`-memprofile`, `top`/`list`/`web` — belongs to **`go-performance` §3**; read it first. This skill is its **deep peer**: the full collection-and-reading workflow, every `go tool pprof` subcommand, the `inuse`-vs-`alloc` heap split, differential profiling, labels, and safe production exposure. *Which* diagnostic to reach for and *how to read a sample statistically* is **`go-perf-methodology`**. Contention (block/mutex/goroutine) → **`go-perf-block-mutex-profiles`**; off-CPU latency/scheduling → **`go-perf-execution-tracer`**; "does it escape" → **`go-perf-escape-analysis`**.

All Go below builds clean under `gofmt`, `go vet`, and `go test` on Go 1.26.

---

## 1. Collecting Profiles — From a Benchmark and From a Server

Two entry points cover almost everything. **From a benchmark** (no code changes, the fastest loop):

```sh
go test -run='^$' -bench=BenchmarkEncode -cpuprofile=cpu.prof -memprofile=mem.prof
go tool pprof cpu.prof                 # interactive REPL
go tool pprof -http=:8080 cpu.prof     # browser UI: flame graph, source, graph
```

**From a live process**, import `net/http/pprof` for its side effect — "The package is typically only imported for the side effect of registering its HTTP handlers" on paths that "all begin with `/debug/pprof/`" ([net/http/pprof](https://pkg.go.dev/net/http/pprof)):

```go
import _ "net/http/pprof" // registers /debug/pprof/ on http.DefaultServeMux

func main() {
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()
	// ... real work ...
}
```

```sh
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30  # 30s CPU
go tool pprof http://localhost:6060/debug/pprof/heap                # live heap
```

`seconds=N` means two different things: for the `profile` (CPU) and `trace` endpoints it sets the sample **duration**; for `allocs`/`block`/`goroutine`/`heap`/`mutex`/`threadcreate` it instead returns a **delta** profile over that window ([net/http/pprof](https://pkg.go.dev/net/http/pprof)) — a built-in differential (§4). **Collect one profile at a time**: "precise memory profiling skews CPU profiles and goroutine blocking profiling affects scheduler trace … it is recommended to collect only a single profile at a time" ([Diagnostics](https://go.dev/doc/diagnostics)).

For a profile without a server, write it by hand — and **`runtime.GC()` first** so live-object stats are current:

```go
runtime.GC() // get up-to-date statistics
if err := pprof.WriteHeapProfile(f); err != nil { // == Lookup("heap").WriteTo(f, 0)
	log.Fatal(err)
}
```

---

## 2. Reading a CPU Profile — Flat vs Cumulative Is the Whole Game

A profile is a **sample**, not ground truth: at ~100 Hz a function with <10 ms total may not appear, and "each stack sample only includes the bottom 100 stack frames," so deep recursion is under-attributed ([pprof](https://go.dev/blog/pprof)). Profile long enough to accumulate samples (the statistical reading is **`go-perf-methodology` §2**).

The subcommands, in the order you actually use them:

| Command | What it shows |
|---|---|
| `top` / `topN` | "the top `N` samples in the profile" — flat cost, heaviest leaves ([pprof](https://go.dev/blog/pprof)) |
| `top -cum` | same list re-sorted by **cumulative** cost — finds expensive *callers* |
| `list Fn` | annotated source, per-line flat/cum counts ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)) |
| `peek Fn` | "the location entry with all its predecessors and successors, without trimming" — callers **and** callees ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)) |
| `tree` | each entry with its predecessors and successors |
| `disasm Fn` | "an annotated disassembly listing" — instruction-level hotspots ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)) |
| `web` / `weblist Fn` | SVG call graph / clickable source→assembly (needs graphviz) ([pprof](https://go.dev/blog/pprof)) |

**Flat vs cumulative is the single most-misread thing in pprof.** The flat columns are "the number of samples in which the function was running (as opposed to waiting for a called function to return)"; the cumulative columns are "the number of samples in which the function appeared (either running or waiting for a called function to return)" ([pprof](https://go.dev/blog/pprof)). In the canonical example `main.FindLoops` "was running in 10.6% of the samples, but it was on the call stack … in 84.1% of the samples" ([pprof](https://go.dev/blog/pprof)).

- **High flat, low cum** → a true leaf hotspot. Optimize *this function's* body.
- **Low flat, high cum** → an expensive *caller*; the cost is in what it calls. `list`/`peek` to find which callee, or do less of the calling.

Default granularity is `-functions`; switch with `-lines`/`-files`/`-addresses` to localize within a big function ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)). Narrow with `-focus=regex`/`-ignore=regex`.

---

## 3. Heap Profiles — `inuse` (Leak) vs `alloc` (Churn)

The heap profile carries **four** views, and picking the wrong one is the most common heap-profiling error. "The heap profile tracks both the allocation sites for all live objects in the application memory and for all objects allocated since the program start. Pprof's `-inuse_space`, `-inuse_objects`, `-alloc_space`, and `-alloc_objects` flags select which to display, defaulting to `-inuse_space` (live objects, scaled by size)" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)).

|  | `*_space` (bytes) | `*_objects` (counts) |
|---|---|---|
| **`inuse_*`** (live now) | live bytes — **hunt a leak / RSS** | live object count — many tiny live objects |
| **`alloc_*`** (cumulative since start) | total bytes ever allocated — **GC churn / pressure** | total allocation count — `allocs/op` at scale |

```sh
go tool pprof -inuse_space mem.prof    # what is resident RIGHT NOW (default)
go tool pprof -alloc_space mem.prof    # what has churned through the GC
go tool pprof -alloc_objects mem.prof  # which sites allocate the most often
```

The split tracks the two questions cleanly:

- **"Memory keeps climbing / OOM"** is a **live-set** question → `inuse_space`. The profile "reports statistics as of the most recently completed garbage collection; it elides more recent allocation to avoid skewing the profile away from live data and toward garbage" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)) — so it shows what *survived*, exactly what a leak is made of.
- **"GC is burning CPU / allocs/op is high"** is a **churn** question → `alloc_space`/`alloc_objects`, "the total number of bytes allocated since the program began (including garbage-collected bytes)" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)). A buffer allocated and freed a million times is invisible to `inuse_*` but dominates `alloc_*`. This is the view that pairs with escape analysis — confirm the site, then **`go-perf-escape-analysis`** explains *why* it heap-allocates.

The memory profiler samples "approximately one block per half megabyte allocated (the '1-in-524288 sampling rate'), so these are approximations" ([pprof](https://go.dev/blog/pprof)) — read magnitudes and rankings, not exact byte counts. `go test -memprofile` and `pprof.Lookup("allocs")` default to `alloc_space`; `Lookup("heap")`/the `/heap` endpoint default to `inuse_space` ([runtime/pprof](https://pkg.go.dev/runtime/pprof)).

---

## 4. Differential Profiling — Comparing Two Profiles

To see what a change (or a leak over time) actually moved, subtract one profile from another "provided the profiles are of compatible types (i.e. two heap profiles)" ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)). Two flags, and they are **not** interchangeable:

```sh
go tool pprof -diff_base=old.prof new.prof   # what got worse/better (percent vs base)
go tool pprof -base=old.prof new.prof        # leak: new live heap minus old live heap
```

- **`-diff_base`** is "useful for comparing two profiles. Percentages in the output are relative to the total of samples in the diff base profile" ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)). Use it for an A/B: hot path before vs after a change. Entries can go **negative** (work that got cheaper).
- **`-base`** is "useful for subtracting a cumulative profile … from another cumulative profile collected from the same program at a later time"; percentages are "relative to the difference between the total for the source profile and the total for the base profile" and all values stay positive ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)). This is the **leak-hunt idiom**: snapshot `inuse_space`, wait, snapshot again, `-base` the first from the second — what remains is what grew.

`-normalize` "scales the source profile so that the total of samples … is equal to the total of samples in the base profile" before subtracting — use it when the two runs had different total durations/load so percentages, not absolute counts, are comparable ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)). (The `/heap?seconds=N` delta endpoint in §1 is a server-side `-base` over a window.)

---

## 5. Profile Labels — Slice the Profile by Request Type

A flat CPU profile blends every code path together; **labels** let you ask "where does the `/checkout` handler spend CPU, separately from `/search`?" Wrap the work in `pprof.Do`, which "calls f with a copy of the parent context with the given labels added … Goroutines spawned while executing f will inherit the augmented label-set" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)):

```go
func handler(w http.ResponseWriter, r *http.Request) {
	labels := pprof.Labels("handler", r.URL.Path, "method", r.Method)
	pprof.Do(r.Context(), labels, func(ctx context.Context) {
		serve(ctx, w, r) // CPU here is tagged; child goroutines inherit the tags
	})
}
```

`pprof.Labels` "takes an even number of strings representing key-value pairs"; crucially "only the CPU and goroutine profiles utilize any labels information" ([runtime/pprof](https://pkg.go.dev/runtime/pprof)) — labels do **not** flow into heap/alloc profiles. Then slice in the tool with `-tagfocus`/`-tagignore`, "the most used option for selecting data in a profile based on tag values," e.g. `-tagfocus=handler=/checkout`, with range syntax like `-tagfocus=128kb:512kb` for numeric tags ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)). Inheritance is the payoff: tag once at the request boundary and every downstream goroutine's CPU is attributed to that request. (`pprof.Do` wraps the lower-level `SetGoroutineLabels`/`WithLabels`; prefer `Do` ([runtime/pprof](https://pkg.go.dev/runtime/pprof)).)

---

## 6. Production & Continuous Profiling — Safely

In-process profiling is cheap enough to run live: "It is safe to profile programs in production, but enabling some profiles (e.g. the CPU profile) adds cost. You should expect to see performance downgrade" ([Diagnostics](https://go.dev/doc/diagnostics)). Two hard rules:

1. **Never expose `/debug/pprof/` on a public listener.** `import _ "net/http/pprof"` registers on `http.DefaultServeMux`; if your app serves business traffic on that same mux, the endpoints (and the stack/heap detail they leak, plus a trivial CPU-burn DoS via `?seconds=`) are internet-reachable. Bind a **separate** admin listener on `localhost`/a private port (the `localhost:6060` goroutine above), or "If you are not using `DefaultServeMux`, you will have to register handlers with the mux you are using" ([net/http/pprof](https://pkg.go.dev/net/http/pprof)) — put them behind auth on an internal-only mux.
2. **One profile at a time** (§1) — concurrent CPU + precise-heap collection skews both ([Diagnostics](https://go.dev/doc/diagnostics)).

**Continuous profiling** turns spot-checks into a time series. A collector scrapes the same `/debug/pprof/` endpoints on a low duty cycle and stores profiles for diffing across deploys. Grafana Pyroscope is "an open source continuous profiling database" that "collects CPU and memory profiles from applications that expose `pprof` endpoints" and keeps "durable, long-term storage … to help you identify changes and trends over time" ([Pyroscope](https://grafana.com/oss/pyroscope/)); Parca and Polar Signals consume the same pprof format. The win is exactly §4 over time: diff today's hot path against last week's, or a regressed pod against a healthy one, instead of trying to reproduce a production hotspot on your laptop.

---

## 7. Don't

- **Don't read flat time when the cost is cumulative** (or vice versa). A low-flat/high-cum function isn't the hotspot — its callee is; use `top -cum`/`peek` ([pprof](https://go.dev/blog/pprof)).
- **Don't debug a leak with `alloc_space`** — churn ≠ live set. A leak lives in `inuse_space`; churn lives in `alloc_*` ([runtime/pprof](https://pkg.go.dev/runtime/pprof)).
- **Don't trust a short profile.** At ~100 Hz a 50 ms benchmark yields ~5 samples — noise; profile longer ([pprof](https://go.dev/blog/pprof)).
- **Don't collect CPU and precise-memory profiles at once** — they skew each other; one at a time ([Diagnostics](https://go.dev/doc/diagnostics)).
- **Don't expose `net/http/pprof` on your public mux** — bind a private admin listener or gate it behind auth ([net/http/pprof](https://pkg.go.dev/net/http/pprof)).
- **Don't expect labels in a heap profile** — only CPU and goroutine profiles carry them ([runtime/pprof](https://pkg.go.dev/runtime/pprof)).
- **Don't confuse `-diff_base` with `-base`** — `-diff_base` for an A/B (percent vs base, negatives allowed), `-base` for subtracting cumulative snapshots (positives, leak hunt) ([pprof README](https://github.com/google/pprof/blob/main/doc/README.md)).
- **Don't forget `runtime.GC()` before a manual heap profile** — without it `inuse_*` includes not-yet-collected garbage ([runtime/pprof](https://pkg.go.dev/runtime/pprof)).
- **Don't reach for pprof to explain *latency* or *blocking*** — a CPU profile is blind to off-CPU waits; that's the tracer and block/mutex profiles.

---

## 8. Routing to Related Skills

**This plugin (`go-perf-*` depth):**

- `go-perf-methodology` — which diagnostic answers which question; reading a profile as a statistical sample; the order of operations.
- `go-perf-block-mutex-profiles` — block/mutex/goroutine profiles and the `SetBlockProfileRate`/`SetMutexProfileFraction` enablement gotcha (off-CPU contention).
- `go-perf-execution-tracer` — `runtime/trace` + `go tool trace` for latency/scheduling/GC the CPU profile can't see; the flight recorder.
- `go-perf-escape-analysis` — once `alloc_*` names a site, `-gcflags=-m` says *why* it heap-allocates.
- `go-perf-gc-tuning` — when the heap profile says GC pressure is the cost.
- `go-perf-pgo` — the CPU profile you collected here *is* the PGO input (`default.pgo`).
- `go-perf-os-tooling` — Linux `perf`/flame graphs when the cost is in the kernel pprof can't attribute.

**Parent / sibling marketplace (`go-*` correctness):**

- `go-performance` — the altitude pprof intro (`top`/`list`/`web`) this skill deepens.
- `go-testing-advanced` — benchmark mechanics that feed `-cpuprofile`/`-memprofile`.

---

## 9. Reference Files

pprof anti-patterns in LLM-generated Go performance work, each with wrong/right contrast and citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)