# Go Perf Os Tooling

> OS-level performance tooling for Go — Linux perf stat (IPC/cache/branch counters) and perf record/report, flame graphs (Gregg's + pprof -http), perf c2c for false-sharing/HITM, strace -c/bpftrace for syscall counts, and perflock for benchmark stability. For off-CPU kernel/hardware cost pprof and the tracer can't see. Fires on "use perf on Go", "cache misses", "flame graph", "perf c2c", "count syscalls". Routes pprof to go-perf-pprof-profiling, false sharing to go-perf-false-sharing, the USE method to go-perf-methodology.

- Skill: `ctoth/go-perf-os-tooling` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add ctoth/go-perf-os-tooling`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ctoth/go-perf-os-tooling/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: ctoth (https://skillmd.com/u/ctoth)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ctoth/go-perf-os-tooling

---


# Go OS-Level Performance Tooling

Go's own diagnostics — pprof, the execution tracer, `runtime/metrics` — see the Go program. They are **blind to the layer below it**: hardware counters (cache misses, branch mispredicts, IPC), kernel/syscall time, and cache-coherence traffic between cores. This skill owns the OS-level instruments — **Linux `perf`**, **flame graphs**, **`perf c2c`**, **bpftrace/bcc**, **`strace`**, and **perflock** — for exactly those cases.

**Honest framing — reach for these last, not first.** If a Go pprof CPU/heap profile, the tracer, or a block/mutex profile can answer the question, use them (they need no root, no kernel config, and attribute Go symbols natively). Drop to OS tooling only when the cost is **off-CPU at the kernel or hardware level**: the CPU profile is hot but IPC is low (memory-bound stalls), a parallel benchmark won't scale (false sharing), or wall time is in syscalls Go's profile can't rank. Pick the *resource* first with the USE method (**`go-perf-methodology`**), then the instrument here.

The symptom-to-instrument map (each row is "Go tools already came up empty"):

| Symptom the Go profiler can't explain | OS instrument | Section |
|---|---|---|
| Hot function, but is it compute- or memory-bound? | `perf stat` (IPC, cache/branch counters) | §1 |
| Time is in the kernel / cgo / native libs | `perf record -g` + `perf report` | §2 |
| Need the profile as one readable picture | flame graph (pprof `-http`, or Gregg's pipeline) | §3 |
| Parallel benchmark won't scale with cores | `perf c2c` (HITM / cache-line contention) | §4 |
| Suspect a syscall storm (one `write` per call) | `strace -c`, bpftrace/bcc | §5 |
| Benchmark numbers swing run-to-run | perflock (serialize + pin the clock) | §6 |

All commands below are shell (gofmt N/A); any Go shown is gofmt-clean on Go 1.25/1.26.

---

## 1. `perf stat` — Hardware Counters pprof Cannot Show

A Go CPU profile tells you *which function* is hot; it cannot tell you *why a cycle was wasted*. `perf stat` reads the CPU's performance-monitoring counters and reports the microarchitectural truth.

```sh
perf stat ./server                    # whole run
perf stat -p $(pgrep server) sleep 30 # attach to a live PID for 30s
perf stat -d ./bench                  # -d adds cache (L1/LLC) detail
```

Counter meanings, verbatim from Gregg ([perf.html](https://www.brendangregg.com/perf.html)):

- **IPC (instructions per cycle):** `8,625,207,199 instructions # 1.53 insns per cycle` — "Higher IPC values mean higher instruction throughput, and lower values indicate more stall cycles." A hot function with IPC well under ~1.0 is **stalled, not computing** — usually waiting on memory; optimize layout/allocation, not the arithmetic.
- **Cache misses:** `L1-dcache-load-misses # 7.96% of all L1-dcache hits` and `LLC-load-misses # 0.00% of all LL-cache hits`. A high LLC-load-miss rate is the cache-unfriendly-data-layout signal (route the fix to `go-perf-data-oriented-layout` / `go-perf-struct-layout`).
- **Branch misses:** `branch-misses # 3.59% of all branches` — prediction accuracy; a high rate points at an unpredictable hot branch.
- **Stall attribution:** `stalled-cycles-frontend # 32.01% frontend cycles idle` and `stalled-cycles-backend # 20.74% backend cycles idle` — "reflecting CPU pipeline bottlenecks."

`perf stat` is the cheapest OS instrument: a single aggregate, no stack walking, no symbols required. Run it **first** to decide whether a hot loop is compute-bound (high IPC — pprof is the right next tool) or memory-bound (low IPC — go look at layout/cache).

---

## 2. `perf record` / `perf report` — Whole-System Sampling

When the bottleneck spans kernel and cgo code a Go profile can't follow, `perf` samples the full stack:

```sh
perf record -F 99 -a -g -- sleep 30   # 99 Hz, all CPUs, with call graphs (-g)
perf report                           # ncurses tree, annotated with percentages
```

`-g` captures call graphs; `perf report` "launches an ncurses navigator for call graph inspection" ([Gregg](https://www.brendangregg.com/perf.html)). **Stacks need frame pointers** — "Always compile with frame pointers. Omitting frame pointers is an evil compiler optimization that breaks debuggers" ([Gregg](https://www.brendangregg.com/perf.html)); Go keeps frame pointers on by default on amd64/arm64, so Go stacks unwind. The alternative is `perf record --call-graph dwarf`.

**Go binaries work with perf.** The Go Diagnostics page: "On Linux, perf tools can be used for profiling Go programs. Perf can profile and unwind cgo/SWIG code and kernel, so it can be useful to get insights into native/kernel performance bottlenecks" ([Diagnostics](https://go.dev/doc/diagnostics)). This is `perf`'s unique value over pprof — it sees **kernel and native time** in the same profile.

**perf ↔ pprof bridge.** You don't have to leave pprof's UI: "pprof can read `perf.data` files generated by the Linux perf tool by using the `perf_to_profile` program from the perf_data_converter package" ([pprof README](https://github.com/google/pprof)). So a `perf record` of a Go server can be explored with the same `go tool pprof` workflow you already know (subcommands → **`go-perf-pprof-profiling`**).

**[VERIFY] async-preemption caveat.** Since Go 1.14 the runtime uses signal-based asynchronous preemption (SIGURG), which can add sampling noise or perturb unwinding under `perf`. The commonly cited mitigation is `GODEBUG=asyncpreemptoff=1` while profiling. The `asyncpreemptoff` GODEBUG value is real (runtime), but the official [Diagnostics](https://go.dev/doc/diagnostics) page does **not** document this as the perf workaround — treat the recommendation as unverified folklore, confirm against your Go version before relying on it, and prefer it only if you see preemption-signal noise.

---

## 3. Flame Graphs — Reading a Profile at a Glance

A flame graph turns thousands of stack samples into one picture. From Gregg ([flamegraphs.html](https://www.brendangregg.com/flamegraphs.html)): "The x-axis shows the stack profile population, sorted alphabetically (**it is not the passage of time**), and the y-axis shows stack depth, counting from zero at the bottom." "**The wider a frame is, the more often it was present in the stacks.**" "Each rectangle represents a stack frame. The top edge shows what is on-CPU, and beneath it is its ancestry." The interactive SVGs let you "Mouse click zooms ... horizontally" and "Search matches and highlights a given term."

**Two ways to get one for Go:**

1. **pprof's built-in flame graph** (no extra tooling — prefer this for pure-Go work). `go tool pprof -http=:8080 cpu.prof` starts the web UI; "If the `-http` flag is specified, pprof starts a web server ... that provides an interactive web-based interface" ([pprof README](https://github.com/google/pprof)). Its Flame Graph view: "Boxes on this view correspond to stack frames in the profile. Caller boxes are directly above callee boxes. The width of each box is proportional to the sum of the sample value of profile samples where that frame was present on the call stack" ([pprof doc/README](https://github.com/google/pprof/blob/main/doc/README.md)). The older `web` command "writes a graph of the profile data in SVG format" ([pprof blog](https://go.dev/blog/pprof)).

2. **Gregg's FlameGraph from `perf` data** (when the kernel/native frames matter). The pipeline folds raw stacks, then renders ([FlameGraph](https://github.com/brendangregg/FlameGraph)):

```sh
perf record -F 99 -a -g -- sleep 60
perf script > out.perf
stackcollapse-perf.pl out.perf > out.folded   # the "folding" step
flamegraph.pl out.folded > out.svg
```

`stackcollapse-perf.pl` collapses each unique stack to one line + a count; `flamegraph.pl` renders the SVG. `go tool pprof -raw` emits pprof's own samples in a text form if you need to bridge formats by hand.

---

## 4. `perf c2c` — False Sharing & Cache-Coherence (HITM)

When a parallel Go benchmark refuses to scale with cores and pprof shows no obvious hot spot, suspect **false sharing**: two cores writing independent variables that share one 64-byte cache line, bouncing the line between caches. `perf c2c` is the instrument that *proves* it.

```sh
perf c2c record -a -- sleep 10
perf c2c report
```

"perf c2c tool provides means for Shared Data C2C/HITM analysis. It allows you to track down the cacheline contentions" — "C2C stands for Cache To Cache" ([perf-c2c man](https://man7.org/linux/man-pages/man1/perf-c2c.1.html)). It ranks "cachelines with highest contention - highest number of **HITM accesses**" (HIT-Modified: a load that hit a cache line another core had modified), split into `HITM - Rmt, Lcl` (remote vs local). A cache line topping the HITM list, with two hot offsets written by different threads, is the false-sharing signature.

The **fix** — pad the contended fields to a cache line (`cpu.CacheLinePad`) — and the full mental model live in **`go-perf-false-sharing`**; `perf c2c` is how you confirm the diagnosis before paying the padding's memory cost.

---

## 5. `strace -c` & bpftrace — Counting and Tracing Syscalls

A Go CPU profile attributes on-CPU time; **syscall time spent blocked in the kernel barely shows up**. To see syscall *volume* — the classic "is my unbuffered writer doing one `write(2)` per call?" question — count them.

**`strace -c`** is the fast confirmation. `strace` is "trace system calls and signals"; `-c` "Counts time, calls, and errors for each system call and report a summary on program exit, suppressing the regular output" ([strace man](https://man7.org/linux/man-pages/man1/strace.1.html)).

```sh
strace -f -c ./server   # -f follows goroutine threads; summary table on exit
```

This is the **proof that a `bufio` fix worked**: count `write` calls before and after wrapping the writer — thousands of small `write`s collapse to a handful. (The buffering technique itself → **`go-perf-buffered-io`**.) Note `strace` adds heavy per-syscall overhead via ptrace, so use it to count, not to time.

**bpftrace** is the low-overhead, production-safe alternative for ad-hoc kernel/user tracing. "bpftrace is a general purpose tracing tool and language for Linux. It leverages eBPF to provide powerful, efficient tracing capabilities with minimal overhead" ([bpftrace](https://github.com/bpftrace/bpftrace)). Count syscalls by process, or histogram latency, with one line ([Gregg](https://www.brendangregg.com/blog/2019-08-19/bpftrace.html)):

```sh
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count() }'
bpftrace -e 'kretprobe:vfs_read { @bytes = lhist(retval, 0, 2000, 200) }'
```

bcc ships the same engine with packaged tools (`syscount`, `funclatency`, `biolatency`). Reach for bpftrace when `strace`'s overhead is unacceptable or the question is kernel-side (off-CPU block latency, I/O size distributions).

---

## 6. perflock — Stabilizing the Benchmark Machine

OS tooling includes making the machine itself a quiet, repeatable instrument — without it, §1–§5 measurements and any benchstat A/B are noise. **perflock** "is a simple locking wrapper for running benchmarks on shared hosts," acquiring "a system-wide lock while running command" ([perflock](https://github.com/aclements/perflock)). Two facts: its **exclusive default** "prevents any other perflock'd command from running" (serializing benches so two never contend for CPU/cache), and **`-governor` (default 90%)** pins "CPU frequency to percent between the min and max," defeating turbo so the clock is constant, then restores the governor on release.

```sh
perflock -governor 90% go test -run='^$' -bench=. -count=10 > new.txt
```

The full A/B method (n≥10, benchstat, p-values) is **`go-perf-benchmarking-statistics`**; perflock is the OS-level half it depends on. Also pin `taskset`/`isolcpus`, disable turbo, fix `GOMAXPROCS` (note container-aware default since Go 1.25).

---

## 7. Routing to Related Skills

- `go-perf-methodology` — pick the saturated *resource* with the USE method **before** reaching for any instrument here; the order of operations.
- `go-perf-pprof-profiling` — the Go-native CPU/heap path you try *first*; pprof subcommands, the `-http` UI, reading an imported `perf.data`.
- `go-perf-execution-tracer` — off-CPU latency/scheduling/GC *inside* Go (the tracer), distinct from kernel-level `perf`.
- `go-perf-false-sharing` — the fix (`cpu.CacheLinePad`, padding) that `perf c2c` (§4) confirms the need for.
- `go-perf-benchmarking-statistics` — benchstat, p-values, n≥10; consumes perflock (§6) as its stabilizer.
- `go-perf-buffered-io` — the `bufio` fix that `strace -c` (§5) proves by syscall count.
- `go-perf-godebug-and-metrics` — the in-process GODEBUG/`runtime/metrics` knobs to try before dropping to the OS.
- `go-perf-data-oriented-layout` / `go-perf-struct-layout` — the cache-friendly layout fix a low-IPC / high-LLC-miss `perf stat` (§1) points to.

---

## 8. Don't

- **Don't reach for `perf` before pprof.** Go's profiler needs no root or kernel config and attributes Go symbols natively; OS tooling is for what pprof is *blind* to — kernel/native time, hardware counters, cache coherence ([Diagnostics](https://go.dev/doc/diagnostics)).
- **Don't optimize a hot function's arithmetic when `perf stat` shows low IPC** — it's stalled on memory, not compute; fix the data layout ([Gregg](https://www.brendangregg.com/perf.html)).
- **Don't read a flame graph's x-axis as time** — it's "sorted alphabetically ... not the passage of time"; width is sample frequency ([Gregg](https://www.brendangregg.com/flamegraphs.html)).
- **Don't blame the algorithm when a parallel bench won't scale** before ruling out false sharing with `perf c2c` HITM analysis ([perf-c2c](https://man7.org/linux/man-pages/man1/perf-c2c.1.html)).
- **Don't use `strace` to *time* a hot path** — its ptrace overhead dominates; use it to *count* syscalls, and bpftrace/bcc when overhead must stay low ([strace](https://man7.org/linux/man-pages/man1/strace.1.html), [bpftrace](https://github.com/bpftrace/bpftrace)).
- **Don't benchmark on an unpinned machine.** Turbo and noisy neighbors bury any real delta; serialize and pin the clock with perflock ([perflock](https://github.com/aclements/perflock)).
- **Don't trust `GODEBUG=asyncpreemptoff=1` as a documented perf fix** — it's unverified against the official docs ([VERIFY], §2); confirm for your Go version before relying on it.

---

## 9. Reference Files

OS-tooling anti-patterns in LLM-generated Go performance work, each with wrong/right contrast and citations:

[references/common-mistakes.md](references/common-mistakes.md)

Source provenance for every claim in this skill:

[references/sources.yaml](references/sources.yaml)