Linux Perf Profiling

Linux Performance Profiling

[GROUNDING-GAP: perf/iostat/sar profiling — grounded on man pages + kernel perf docs; deepen with Systems Performance 2e and BPF Performance Tools (Brendan Gregg).]

Distro support

One-family-pair skill: the tools below behave identically on Debian/Ubuntu and the RHEL family (Fedora, RHEL, CentOS Stream, Rocky, Alma, Oracle). The /proc-backed tools (uptime, top, vmstat, free, ps) ship in procps (Debian) / procps-ng (RHEL) and are usually preinstalled. The sysstat tools (iostat, mpstat, pidstat, sar) come from the sysstat package on both families. perf is the only tool whose package name differs.

htop is in the default repos on both (apt install htop / dnf install htop); on minimal RHEL it may need EPEL on older releases.

Use When

• A host is "slow" and you must locate the bottleneck before changing anything.
• Deciding whether the limit is CPU, disk I/O, or memory before tuning.
• You need a CPU profile (which functions burn cycles) via perf.
• You want historical context (sar) for an incident that already happened.

Do Not Use When

• You already know the bottleneck and only need to apply kernel tunables; use linux-sysctl-tuning.
• The question is per-service CPU/memory share or limits; use linux-service-priority (cgroups).
• You are tracing application logic, not system resources (use app-level APM).

Required Inputs

• The symptom (latency, throughput drop, OOM, high load) and when it occurs.
• Whether you can install packages (sysstat, perf) on the host.
• Whether the issue is live (sample now) or past (need sar history).

Methodology (USE)

Brendan Gregg's USE method — for each resource, check Utilization, Saturation, Errors — keeps profiling systematic instead of guessing:

Work top-down: read load average, then split CPU vs I/O-wait vs memory with one vmstat 1 sample, then drill into the saturated resource with the specific tool. Measure first; tune second (then hand off to linux-sysctl-tuning).

Capability Contract

Default to read-only diagnosis and search of available metrics. Process/host sampling must be bounded. Package installation, lowering perf_event_paranoid, workload generation, tuning, restarts, and configuration changes require explicit permission; this skill does not tune.

Degraded Mode

When perf, sysstat history, kernel symbols, or target access is unavailable, use the narrowest available USE evidence and label missing dimensions not assessed. Never convert one instantaneous sample into a pass or root-cause claim.

Decision Rules

Workflow

1. Triage: uptime (load vs core count), then vmstat 1 for a few lines.

2. Classify: high wa → disk; high r with low idle → CPU; high si/so or low available memory → memory pressure (see decision table below).

3. Drill: disk → iostat -x 1; CPU → mpstat -P ALL 1 + pidstat 1; memory → free -h, vmstat, dmesg | grep -i oom.

4. Profile CPU (if CPU-bound): perf top, then perf record -g + perf report to see the hot stacks.

5. Conclude: name the bottleneck and the evidence; hand off to a tuning skill rather than tuning blind here.

6. Stop sampling if overhead, output growth, or scope exceeds the bound; recover by ending the profiler, preserving partial evidence, and fall back to lower-overhead timed metrics.

Quality Standards

• Sample over a window (vmstat 1, several lines) — never trust one instant.
• Ignore the first iostat/mpstat line: it is averages since boot, not now.
• Pair every claim with the field that proves it (e.g. "disk-bound: %util 98, await 40 ms").

Anti-Patterns

• Tuning before measurement. Fix: finish bounded USE triage and name the supported bottleneck.

• Reading the first since-boot sample as current. Fix: discard it and record a timed window.

• Equating load with CPU. Fix: separate runnable tasks from I/O-wait and process evidence.

• Running unbounded system-wide perf. Fix: cap frequency, duration, scope, and output size.

• Inventing root cause when the symptom is absent. Fix: correlate history and report inconclusive.

• Tuning sysctl/storage before measuring the actual bottleneck.

• Reading the since-boot first sample as the current state.

• Treating high load average as automatically CPU — on Linux load includes uninterruptible (D-state, usually I/O) tasks.

• Running perf record system-wide for minutes on a busy box without bounding it (-F frequency, a fixed duration) — the perf.data file balloons.

• Declaring root cause from correlation alone. Correction: reproduce or triangulate with resource and process evidence.

• Omitting the sampling window and workload. Correction: record duration, interval, scope, and request rate/concurrency.

Outputs

Evidence Produced

Capture timestamped bounded samples, tool versions, workload context, historical comparison, perf command/overhead, top stacks where available, errors, and an explicit inconclusive result when evidence does not isolate a bottleneck.

Worked Example

For a latency spike, record request load, sample vmstat, ignore the first iostat line, correlate high device await with per-process I/O and kernel errors, then hand storage evidence to the owning team. High load alone is insufficient.

• The classified bottleneck (CPU / I/O-wait / memory) and the metrics proving it.
• For CPU cases, the top symbols/stacks from perf report.
• A concrete next step (which tuning skill, which subsystem).

Triage: classify the bottleneck

Take one vmstat 1 window and read three columns:

vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

#CPUs = nproc. Load average above nproc is normal only if the extra tasks are runnable (CPU) — if they are in D-state (ps -eo stat), it is I/O.

Live tools

uptime                       # 1/5/15-min load avg; compare to `nproc`
top                          # press '1' for per-CPU, 'P'/'M' sort by CPU/MEM
htop                         # nicer top: per-core bars, tree view, F6 sort
vmstat 1                     # virtual-mem/CPU/IO sampling (ignore line 1)
free -h                      # memory: look at 'available', not 'free'

Disk I/O — iostat -x

iostat -x 1                  # extended per-device stats (ignore first sample)

Read these fields per device:

• %util — fraction of time the device had I/O in flight. Near 100% means the device is saturated (caveat: for SSD/NVMe with parallel queues, %util can hit 100% while the device still has headroom — corroborate with await).
• await — average ms per I/O (queue + service time). Rising await under load is the clearest saturation signal; spinning disks ~5–15 ms healthy, tens-to-hundreds ms = backed up.
• r/s w/s and rkB/s wkB/s — IOPS and throughput, to see whether the load is random (high IOPS) or sequential (high kB/s).
• aqu-sz (avg queue length) — sustained > 1 means requests are queuing.

A disk bottleneck is high %util AND high await together. Then find the culprit process with pidstat -d 1.

Per-CPU and per-process — mpstat / pidstat

mpstat -P ALL 1              # per-core %usr/%sys/%iowait/%idle — spot a hot core
pidstat 1                    # per-process %CPU each interval
pidstat -d 1                 # per-process disk read/write KB/s
pidstat -r 1                 # per-process memory faults / RSS

One core pinned at 100% while others idle = a single-threaded hot path (mpstat exposes this; top's aggregate hides it).

Historical — sar (sysstat)

sar reads the data the sysstat collector records on a schedule, so you can look back at an incident:

sar                          # today's CPU history (default)
sar -u                       # CPU; sar -r memory; sar -b I/O; sar -n DEV network
sar -d -p                    # per-device disk (pretty device names)
sar -f /var/log/sysstat/saYYMMDD   # a specific past day's file

Enable collection once: install sysstat, then enable the collector (systemctl enable --now sysstat and, on most distros, sysstat-collect.timer). Without it, sar has no history.

CPU profiling with perf

perf samples the CPU and attributes time to functions — it answers which code is hot, which top/mpstat cannot.

# Install: Debian -> linux-perf OR linux-tools-$(uname -r) + linux-tools-common
#          RHEL   -> dnf install perf

perf top                              # live, top symbols by CPU (like top, per-fn)
perf stat -- <cmd>                    # counters for one command: IPC, cache, ctx-sw
perf stat -a sleep 5                  # system-wide counters for 5s

# Sampled profile with call graphs:
perf record -g -- <cmd>               # profile one command (-g = call stacks)
perf record -g -p <PID> -- sleep 30   # attach to a running PID for 30s
perf record -g -a -- sleep 30         # whole system for 30s (bound the duration!)
perf report                           # interactive browser of perf.data
perf report --stdio | head -40        # non-interactive top stacks

Read perf stat for cheap insight first: IPC (instructions/cycle) below ~1 suggests stalls (memory-bound); high context-switches suggests scheduling contention; high cache-misses points at memory layout.

kernel.perf_event_paranoid

The kernel gates non-root perf access via this sysctl. Values:

sysctl kernel.perf_event_paranoid          # check current
sudo sysctl -w kernel.perf_event_paranoid=1   # transient; for kernel symbols

Lowering it system-wide weakens isolation — prefer running perf as root for a one-off, or persist a relaxed value only on a trusted profiling host. See linux-sysctl-tuning for the drop-in persistence pattern.

Flame graphs

Turn a perf record into a flame graph (visual stack census) with Brendan Gregg's FlameGraph scripts — see references/profiling-tools.md for the pipe.

Scripts

This skill installs the following scripts to /usr/local/bin/. To install:

sudo install-skills-bin linux-perf-profiling

References

• references/profiling-tools.md