Linux perf
perf ships in the kernel source tree under tools/perf, so its feature set follows the running kernel line (mainline 7.2, LTS 6.18). This skill covers the three jobs perf does for a CPU-bound program: count hardware events with perf stat, sample where cycles go with perf record and perf report, and attribute samples to instructions with perf annotate.
Contract
| Field |
Bound contract |
| Trigger |
A Linux program is slow or CPU-bound and the user wants hot functions, counter totals, or instruction-level attribution from perf. Also fires on [unknown] frames, Permission denied from perf, or a request to feed perf data to a flamegraph pipeline. |
| Authority |
Reversible local: writes only perf.data files (or the -o target) and the compiled test binary in the working directory; rollback is deleting those files. A sysctl change to kernel.perf_event_paranoid is proposed to the user, never applied by the skill. No remote mutation. |
| Side effect |
Profile data files on disk. The profiled program runs under sampling, which slows it by a small factor that depends on sample frequency and call-graph mode. |
| Done |
The report names the hot symbols with their sample share, every counter reading carries the workload condition it was taken under, and any unresolved frame is explained with its fix. |
Inputs
- Target program and a representative workload (required). Short programs need a longer input or a higher
-F.
- Debug symbols (required for symbol names and source lines): build with
-g. For frame-pointer call graphs add -fno-omit-frame-pointer; otherwise use --call-graph dwarf.
- Value of
/proc/sys/kernel/perf_event_paranoid (gathered by the skill). Meaning: 2 allows user-space measurement of the caller's own processes only, 1 adds kernel measurement, 0 adds CPU-wide events, -1 removes all restrictions. Values above 2 block unprivileged perf entirely.
- A question: which function is hot, which counter is high, or which instruction stalls. The question picks the subcommand.
Procedure
- Check permissions. Read
cat /proc/sys/kernel/perf_event_paranoid. If the value blocks the intended measurement, propose sudo sysctl -w kernel.perf_event_paranoid=1 for the session, or the persistent form echo 'kernel.perf_event_paranoid=1' | sudo tee /etc/sysctl.d/99-perf.conf followed by sudo sysctl -p /etc/sysctl.d/99-perf.conf. Wait for the user to apply it. Done when: perf can open the events the plan needs, or the user has declined and the plan is narrowed to what the current level allows.
- Build with symbols:
gcc -g -O2 -fno-omit-frame-pointer -o prog main.c. Keep the optimization level of the real build, because a -O0 profile points at code that does not exist in production. Done when: nm prog | head shows symbols and the binary runs the workload.
- Count first with
perf stat. Run perf stat ./prog for the default set, then perf stat -e cycles,instructions,cache-references,cache-misses,branches,branch-misses ./prog for named events, and perf stat -r 5 ./prog to average five runs and print the spread. Attach to a live process with perf stat -p <pid> sleep 10. Read derived ratios as conditional signals, not verdicts: instructions per cycle depends on the core's issue width and on whether the workload is memory-bound by design, so compare IPC between two builds of the same program on the same machine rather than against a fixed number. A miss rate matters only when the absolute miss count is large relative to runtime. Done when: the counter table and the run-to-run spread are recorded with the workload name.
- Sample with
perf record. Use perf record -F 999 -g ./prog for a frame-pointer call graph at 999 Hz (an odd frequency avoids lock-step with periodic timers). Use perf record -F 999 --call-graph dwarf ./prog when the binary or its libraries lack frame pointers; cap the copied stack with --call-graph dwarf,4096 if the data file grows too fast. Sample a specific event with -e cache-misses, attach with -p <pid> sleep 30, sample all CPUs with -a, and name the output with -o app.perf.data. Done when: perf record prints the sample count and the data file exists.
- Read the report.
perf report opens the TUI on perf.data; perf report -i app.perf.data reads a named file; perf report --stdio prints text; perf report --no-children shows self time instead of inclusive time; perf report --sort comm,dso,sym groups by process, library, and symbol. In the TUI, Enter expands a symbol, a annotates it, d filters by DSO, t filters by thread, and ? lists keys. Done when: the top symbols by self time are listed with their percentages.
- Annotate the hot symbol.
perf annotate -i perf.data --symbol=<name> --stdio prints the disassembly with per-instruction sample shares; s in the TUI toggles source when debug info is present. A high share on a load instruction (mov, vmovdqa) marks a stall on that load, usually a cache miss. Done when: the hottest instructions and their source lines are named.
- Watch live when the workload is long-running:
sudo perf top -g or sudo perf top -p <pid>. Done when: the live view confirms or refutes the recorded hotspot.
- Export for a flamegraph:
perf script -i perf.data > out.perf, then hand out.perf to flamegraphs (stackcollapse-perf.pl and flamegraph.pl). Done when: out.perf exists.
- Resolve any bad frames or empty output using the failure table below before reporting. Done when: no
[unknown] frame remains in the top entries, or its cause is stated.
Event names to reach for: hardware cycles, instructions, cache-references, cache-misses, branches, branch-misses, stalled-cycles-frontend, stalled-cycles-backend; cache L1-dcache-loads, L1-dcache-load-misses, LLC-loads, LLC-load-misses, dTLB-load-misses; software context-switches, cpu-migrations, page-faults, major-faults; tracepoints such as sched:sched_switch (need paranoid level 0 or root). perf list prints what this CPU exposes; perf list hw, perf list cache, perf list sw, and perf list tracepoint filter by class. Raw PMU codes are CPU-specific: perf list pmu names them, and the form perf stat -e cpu/event=0xd1,umask=0x20/u passes one by code. Full event notes live in references/events.md.
Failure and recovery
| Failure |
Cause |
Fix |
Permission denied or Error: Access to performance monitoring and observability operations is not permitted |
perf_event_paranoid above the needed level |
Propose the sysctl change from step 1, or run the measurement with sudo |
[unknown] frames |
No frame pointers or no debug info |
Rebuild with -g -fno-omit-frame-pointer, or record with --call-graph dwarf |
| Kernel frames show as addresses |
Kernel symbols hidden |
Record with sudo; kptr_restrict at 0 (/proc/sys/kernel/kptr_restrict) exposes /proc/kallsyms; install the kernel debug symbols package for the running kernel |
| Empty report for a short program |
Too few samples |
Raise -F toward the value in kernel.perf_event_max_sample_rate, or lengthen the workload |
| DWARF unwinding is slow or the data file is huge |
Whole user stack copied per sample |
--call-graph dwarf,4096 or a smaller size, or add frame pointers and use -g |
Counter shows <not supported> |
Event absent on this CPU or virtualized |
Pick a name from perf list; in a VM only software events may exist |
A partial profile is reported as partial: the report states which steps ran and which measurement is missing.
Output
A profile report containing the counter table from step 3 with its run spread and workload name, the top symbols by self time from step 5 with percentages, the hot instructions and source lines from step 6, the path of every data file written, and any unresolved frame with its cause and fix.
1---2name: linux-perf3description: Use when collecting sampling profiles with perf record, reading perf report or perf annotate, or measuring counters with perf stat on Linux. Not for SVG rendering: use flamegraphs.4---56# Linux perf78`perf` ships in the kernel source tree under `tools/perf`, so its feature set follows the running kernel line (mainline 7.2, LTS 6.18). This skill covers the three jobs perf does for a CPU-bound program: count hardware events with `perf stat`, sample where cycles go with `perf record` and `perf report`, and attribute samples to instructions with `perf annotate`.910## Contract1112| Field | Bound contract |13|---|---|14| Trigger | A Linux program is slow or CPU-bound and the user wants hot functions, counter totals, or instruction-level attribution from `perf`. Also fires on `[unknown]` frames, `Permission denied` from perf, or a request to feed perf data to a flamegraph pipeline. |15| Authority | Reversible local: writes only `perf.data` files (or the `-o` target) and the compiled test binary in the working directory; rollback is deleting those files. A `sysctl` change to `kernel.perf_event_paranoid` is proposed to the user, never applied by the skill. No remote mutation. |16| Side effect | Profile data files on disk. The profiled program runs under sampling, which slows it by a small factor that depends on sample frequency and call-graph mode. |17| Done | The report names the hot symbols with their sample share, every counter reading carries the workload condition it was taken under, and any unresolved frame is explained with its fix. |1819## Inputs2021- Target program and a representative workload (required). Short programs need a longer input or a higher `-F`.22- Debug symbols (required for symbol names and source lines): build with `-g`. For frame-pointer call graphs add `-fno-omit-frame-pointer`; otherwise use `--call-graph dwarf`.23- Value of `/proc/sys/kernel/perf_event_paranoid` (gathered by the skill). Meaning: `2` allows user-space measurement of the caller's own processes only, `1` adds kernel measurement, `0` adds CPU-wide events, `-1` removes all restrictions. Values above `2` block unprivileged perf entirely.24- A question: which function is hot, which counter is high, or which instruction stalls. The question picks the subcommand.2526## Procedure27281. Check permissions. Read `cat /proc/sys/kernel/perf_event_paranoid`. If the value blocks the intended measurement, propose `sudo sysctl -w kernel.perf_event_paranoid=1` for the session, or the persistent form `echo 'kernel.perf_event_paranoid=1' | sudo tee /etc/sysctl.d/99-perf.conf` followed by `sudo sysctl -p /etc/sysctl.d/99-perf.conf`. Wait for the user to apply it. Done when: perf can open the events the plan needs, or the user has declined and the plan is narrowed to what the current level allows.292. Build with symbols: `gcc -g -O2 -fno-omit-frame-pointer -o prog main.c`. Keep the optimization level of the real build, because a `-O0` profile points at code that does not exist in production. Done when: `nm prog | head` shows symbols and the binary runs the workload.303. Count first with `perf stat`. Run `perf stat ./prog` for the default set, then `perf stat -e cycles,instructions,cache-references,cache-misses,branches,branch-misses ./prog` for named events, and `perf stat -r 5 ./prog` to average five runs and print the spread. Attach to a live process with `perf stat -p <pid> sleep 10`. Read derived ratios as conditional signals, not verdicts: instructions per cycle depends on the core's issue width and on whether the workload is memory-bound by design, so compare IPC between two builds of the same program on the same machine rather than against a fixed number. A miss rate matters only when the absolute miss count is large relative to runtime. Done when: the counter table and the run-to-run spread are recorded with the workload name.314. Sample with `perf record`. Use `perf record -F 999 -g ./prog` for a frame-pointer call graph at 999 Hz (an odd frequency avoids lock-step with periodic timers). Use `perf record -F 999 --call-graph dwarf ./prog` when the binary or its libraries lack frame pointers; cap the copied stack with `--call-graph dwarf,4096` if the data file grows too fast. Sample a specific event with `-e cache-misses`, attach with `-p <pid> sleep 30`, sample all CPUs with `-a`, and name the output with `-o app.perf.data`. Done when: `perf record` prints the sample count and the data file exists.325. Read the report. `perf report` opens the TUI on `perf.data`; `perf report -i app.perf.data` reads a named file; `perf report --stdio` prints text; `perf report --no-children` shows self time instead of inclusive time; `perf report --sort comm,dso,sym` groups by process, library, and symbol. In the TUI, `Enter` expands a symbol, `a` annotates it, `d` filters by DSO, `t` filters by thread, and `?` lists keys. Done when: the top symbols by self time are listed with their percentages.336. Annotate the hot symbol. `perf annotate -i perf.data --symbol=<name> --stdio` prints the disassembly with per-instruction sample shares; `s` in the TUI toggles source when debug info is present. A high share on a load instruction (`mov`, `vmovdqa`) marks a stall on that load, usually a cache miss. Done when: the hottest instructions and their source lines are named.347. Watch live when the workload is long-running: `sudo perf top -g` or `sudo perf top -p <pid>`. Done when: the live view confirms or refutes the recorded hotspot.358. Export for a flamegraph: `perf script -i perf.data > out.perf`, then hand `out.perf` to `flamegraphs` (`stackcollapse-perf.pl` and `flamegraph.pl`). Done when: `out.perf` exists.369. Resolve any bad frames or empty output using the failure table below before reporting. Done when: no `[unknown]` frame remains in the top entries, or its cause is stated.3738Event names to reach for: hardware `cycles`, `instructions`, `cache-references`, `cache-misses`, `branches`, `branch-misses`, `stalled-cycles-frontend`, `stalled-cycles-backend`; cache `L1-dcache-loads`, `L1-dcache-load-misses`, `LLC-loads`, `LLC-load-misses`, `dTLB-load-misses`; software `context-switches`, `cpu-migrations`, `page-faults`, `major-faults`; tracepoints such as `sched:sched_switch` (need paranoid level 0 or root). `perf list` prints what this CPU exposes; `perf list hw`, `perf list cache`, `perf list sw`, and `perf list tracepoint` filter by class. Raw PMU codes are CPU-specific: `perf list pmu` names them, and the form `perf stat -e cpu/event=0xd1,umask=0x20/u` passes one by code. Full event notes live in `references/events.md`.3940## Failure and recovery4142| Failure | Cause | Fix |43|---|---|---|44| `Permission denied` or `Error: Access to performance monitoring and observability operations is not permitted` | `perf_event_paranoid` above the needed level | Propose the `sysctl` change from step 1, or run the measurement with `sudo` |45| `[unknown]` frames | No frame pointers or no debug info | Rebuild with `-g -fno-omit-frame-pointer`, or record with `--call-graph dwarf` |46| Kernel frames show as addresses | Kernel symbols hidden | Record with `sudo`; `kptr_restrict` at `0` (`/proc/sys/kernel/kptr_restrict`) exposes `/proc/kallsyms`; install the kernel debug symbols package for the running kernel |47| Empty report for a short program | Too few samples | Raise `-F` toward the value in `kernel.perf_event_max_sample_rate`, or lengthen the workload |48| DWARF unwinding is slow or the data file is huge | Whole user stack copied per sample | `--call-graph dwarf,4096` or a smaller size, or add frame pointers and use `-g` |49| Counter shows `<not supported>` | Event absent on this CPU or virtualized | Pick a name from `perf list`; in a VM only software events may exist |5051A partial profile is reported as partial: the report states which steps ran and which measurement is missing.5253## Output5455A profile report containing the counter table from step 3 with its run spread and workload name, the top symbols by self time from step 5 with percentages, the hot instructions and source lines from step 6, the path of every data file written, and any unresolved frame with its cause and fix.
OutlineDriven