Go Perf Os Tooling

Go OS-Level Performance Tooling

Go's own diagnostics — pprof, the execution tracer, runtime/metrics — see the Go program. They are blind to the layer below it: hardware counters (cache misses, branch mispredicts, IPC), kernel/syscall time, and cache-coherence traffic between cores. This skill owns the OS-level instruments — Linux perf, flame graphs, perf c2c, bpftrace/bcc, strace, and perflock — for exactly those cases.

Honest framing — reach for these last, not first. If a Go pprof CPU/heap profile, the tracer, or a block/mutex profile can answer the question, use them (they need no root, no kernel config, and attribute Go symbols natively). Drop to OS tooling only when the cost is off-CPU at the kernel or hardware level: the CPU profile is hot but IPC is low (memory-bound stalls), a parallel benchmark won't scale (false sharing), or wall time is in syscalls Go's profile can't rank. Pick the resource first with the USE method (go-perf-methodology), then the instrument here.

The symptom-to-instrument map (each row is "Go tools already came up empty"):

All commands below are shell (gofmt N/A); any Go shown is gofmt-clean on Go 1.25/1.26.

1. perf stat — Hardware Counters pprof Cannot Show

A Go CPU profile tells you which function is hot; it cannot tell you why a cycle was wasted. perf stat reads the CPU's performance-monitoring counters and reports the microarchitectural truth.

perf stat ./server                    # whole run
perf stat -p $(pgrep server) sleep 30 # attach to a live PID for 30s
perf stat -d ./bench                  # -d adds cache (L1/LLC) detail

Counter meanings, verbatim from Gregg (perf.html):

• IPC (instructions per cycle): 8,625,207,199 instructions # 1.53 insns per cycle — "Higher IPC values mean higher instruction throughput, and lower values indicate more stall cycles." A hot function with IPC well under ~1.0 is stalled, not computing — usually waiting on memory; optimize layout/allocation, not the arithmetic.
• Cache misses: L1-dcache-load-misses # 7.96% of all L1-dcache hits and LLC-load-misses # 0.00% of all LL-cache hits. A high LLC-load-miss rate is the cache-unfriendly-data-layout signal (route the fix to go-perf-data-oriented-layout / go-perf-struct-layout).
• Branch misses: branch-misses # 3.59% of all branches — prediction accuracy; a high rate points at an unpredictable hot branch.
• Stall attribution: stalled-cycles-frontend # 32.01% frontend cycles idle and stalled-cycles-backend # 20.74% backend cycles idle — "reflecting CPU pipeline bottlenecks."

perf stat is the cheapest OS instrument: a single aggregate, no stack walking, no symbols required. Run it first to decide whether a hot loop is compute-bound (high IPC — pprof is the right next tool) or memory-bound (low IPC — go look at layout/cache).

2. perf record / perf report — Whole-System Sampling

When the bottleneck spans kernel and cgo code a Go profile can't follow, perf samples the full stack:

perf record -F 99 -a -g -- sleep 30   # 99 Hz, all CPUs, with call graphs (-g)
perf report                           # ncurses tree, annotated with percentages

-g captures call graphs; perf report "launches an ncurses navigator for call graph inspection" (Gregg). Stacks need frame pointers — "Always compile with frame pointers. Omitting frame pointers is an evil compiler optimization that breaks debuggers" (Gregg); Go keeps frame pointers on by default on amd64/arm64, so Go stacks unwind. The alternative is perf record --call-graph dwarf.

Go binaries work with perf. The Go Diagnostics page: "On Linux, perf tools can be used for profiling Go programs. Perf can profile and unwind cgo/SWIG code and kernel, so it can be useful to get insights into native/kernel performance bottlenecks" (Diagnostics). This is perf's unique value over pprof — it sees kernel and native time in the same profile.

perf ↔ pprof bridge. You don't have to leave pprof's UI: "pprof can read perf.data files generated by the Linux perf tool by using the perf_to_profile program from the perf_data_converter package" (pprof README). So a perf record of a Go server can be explored with the same go tool pprof workflow you already know (subcommands → go-perf-pprof-profiling).

[VERIFY] async-preemption caveat. Since Go 1.14 the runtime uses signal-based asynchronous preemption (SIGURG), which can add sampling noise or perturb unwinding under perf. The commonly cited mitigation is GODEBUG=asyncpreemptoff=1 while profiling. The asyncpreemptoff GODEBUG value is real (runtime), but the official Diagnostics page does not document this as the perf workaround — treat the recommendation as unverified folklore, confirm against your Go version before relying on it, and prefer it only if you see preemption-signal noise.

3. Flame Graphs — Reading a Profile at a Glance

A flame graph turns thousands of stack samples into one picture. From Gregg (flamegraphs.html): "The x-axis shows the stack profile population, sorted alphabetically (it is not the passage of time), and the y-axis shows stack depth, counting from zero at the bottom." "The wider a frame is, the more often it was present in the stacks." "Each rectangle represents a stack frame. The top edge shows what is on-CPU, and beneath it is its ancestry." The interactive SVGs let you "Mouse click zooms ... horizontally" and "Search matches and highlights a given term."

Two ways to get one for Go:

1. pprof's built-in flame graph (no extra tooling — prefer this for pure-Go work). go tool pprof -http=:8080 cpu.prof starts the web UI; "If the -http flag is specified, pprof starts a web server ... that provides an interactive web-based interface" (pprof README). Its Flame Graph view: "Boxes on this view correspond to stack frames in the profile. Caller boxes are directly above callee boxes. The width of each box is proportional to the sum of the sample value of profile samples where that frame was present on the call stack" (pprof doc/README). The older web command "writes a graph of the profile data in SVG format" (pprof blog).

2. Gregg's FlameGraph from perf data (when the kernel/native frames matter). The pipeline folds raw stacks, then renders (FlameGraph):

perf record -F 99 -a -g -- sleep 60
perf script > out.perf
stackcollapse-perf.pl out.perf > out.folded   # the "folding" step
flamegraph.pl out.folded > out.svg

stackcollapse-perf.pl collapses each unique stack to one line + a count; flamegraph.pl renders the SVG. go tool pprof -raw emits pprof's own samples in a text form if you need to bridge formats by hand.

4. perf c2c — False Sharing & Cache-Coherence (HITM)

When a parallel Go benchmark refuses to scale with cores and pprof shows no obvious hot spot, suspect false sharing: two cores writing independent variables that share one 64-byte cache line, bouncing the line between caches. perf c2c is the instrument that proves it.

perf c2c record -a -- sleep 10
perf c2c report

"perf c2c tool provides means for Shared Data C2C/HITM analysis. It allows you to track down the cacheline contentions" — "C2C stands for Cache To Cache" (perf-c2c man). It ranks "cachelines with highest contention - highest number of HITM accesses" (HIT-Modified: a load that hit a cache line another core had modified), split into HITM - Rmt, Lcl (remote vs local). A cache line topping the HITM list, with two hot offsets written by different threads, is the false-sharing signature.

The fix — pad the contended fields to a cache line (cpu.CacheLinePad) — and the full mental model live in go-perf-false-sharing; perf c2c is how you confirm the diagnosis before paying the padding's memory cost.

5. strace -c & bpftrace — Counting and Tracing Syscalls

A Go CPU profile attributes on-CPU time; syscall time spent blocked in the kernel barely shows up. To see syscall volume — the classic "is my unbuffered writer doing one write(2) per call?" question — count them.

strace -c is the fast confirmation. strace is "trace system calls and signals"; -c "Counts time, calls, and errors for each system call and report a summary on program exit, suppressing the regular output" (strace man).

strace -f -c ./server   # -f follows goroutine threads; summary table on exit

This is the proof that a bufio fix worked: count write calls before and after wrapping the writer — thousands of small writes collapse to a handful. (The buffering technique itself → go-perf-buffered-io.) Note strace adds heavy per-syscall overhead via ptrace, so use it to count, not to time.

bpftrace is the low-overhead, production-safe alternative for ad-hoc kernel/user tracing. "bpftrace is a general purpose tracing tool and language for Linux. It leverages eBPF to provide powerful, efficient tracing capabilities with minimal overhead" (bpftrace). Count syscalls by process, or histogram latency, with one line (Gregg):

bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count() }'
bpftrace -e 'kretprobe:vfs_read { @bytes = lhist(retval, 0, 2000, 200) }'

bcc ships the same engine with packaged tools (syscount, funclatency, biolatency). Reach for bpftrace when strace's overhead is unacceptable or the question is kernel-side (off-CPU block latency, I/O size distributions).

6. perflock — Stabilizing the Benchmark Machine

OS tooling includes making the machine itself a quiet, repeatable instrument — without it, §1–§5 measurements and any benchstat A/B are noise. perflock "is a simple locking wrapper for running benchmarks on shared hosts," acquiring "a system-wide lock while running command" (perflock). Two facts: its exclusive default "prevents any other perflock'd command from running" (serializing benches so two never contend for CPU/cache), and -governor (default 90%) pins "CPU frequency to percent between the min and max," defeating turbo so the clock is constant, then restores the governor on release.

perflock -governor 90% go test -run='^$' -bench=. -count=10 > new.txt

The full A/B method (n≥10, benchstat, p-values) is go-perf-benchmarking-statistics; perflock is the OS-level half it depends on. Also pin taskset/isolcpus, disable turbo, fix GOMAXPROCS (note container-aware default since Go 1.25).

7. Routing to Related Skills

• go-perf-methodology — pick the saturated resource with the USE method before reaching for any instrument here; the order of operations.
• go-perf-pprof-profiling — the Go-native CPU/heap path you try first; pprof subcommands, the -http UI, reading an imported perf.data.
• go-perf-execution-tracer — off-CPU latency/scheduling/GC inside Go (the tracer), distinct from kernel-level perf.
• go-perf-false-sharing — the fix (cpu.CacheLinePad, padding) that perf c2c (§4) confirms the need for.
• go-perf-benchmarking-statistics — benchstat, p-values, n≥10; consumes perflock (§6) as its stabilizer.
• go-perf-buffered-io — the bufio fix that strace -c (§5) proves by syscall count.
• go-perf-godebug-and-metrics — the in-process GODEBUG/runtime/metrics knobs to try before dropping to the OS.
• go-perf-data-oriented-layout / go-perf-struct-layout — the cache-friendly layout fix a low-IPC / high-LLC-miss perf stat (§1) points to.

8. Don't

• Don't reach for perf before pprof. Go's profiler needs no root or kernel config and attributes Go symbols natively; OS tooling is for what pprof is blind to — kernel/native time, hardware counters, cache coherence (Diagnostics).
• Don't optimize a hot function's arithmetic when perf stat shows low IPC — it's stalled on memory, not compute; fix the data layout (Gregg).
• Don't read a flame graph's x-axis as time — it's "sorted alphabetically ... not the passage of time"; width is sample frequency (Gregg).
• Don't blame the algorithm when a parallel bench won't scale before ruling out false sharing with perf c2c HITM analysis (perf-c2c).
• Don't use strace to time a hot path — its ptrace overhead dominates; use it to count syscalls, and bpftrace/bcc when overhead must stay low (strace, bpftrace).
• Don't benchmark on an unpinned machine. Turbo and noisy neighbors bury any real delta; serialize and pin the clock with perflock (perflock).
• Don't trust GODEBUG=asyncpreemptoff=1 as a documented perf fix — it's unverified against the official docs ([VERIFY], §2); confirm for your Go version before relying on it.

9. Reference Files

OS-tooling anti-patterns in LLM-generated Go performance work, each with wrong/right contrast and citations:

references/common-mistakes.md

Source provenance for every claim in this skill:

references/sources.yaml