# Brendangregg Use Tsa

> Methodical system performance troubleshooting and root cause analysis using Brendan Gregg's USE Method (Utilization, Saturation, Errors) and TSA Method (Thread State Analysis), plus off-CPU analysis, flame graphs, and structured reporting. Use when investigating slow systems, latency, high CPU/memory/disk/network usage, bottlenecks, hangs, throughput drops, or mysterious resource exhaustion on any OS; when asked to debug, triage, or root-cause a performance issue; and when writing RCA reports or postmortems.

- Skill: `thecsdoctor/brendangregg-use-tsa-2` (Agent Skill, multi-file: 9 files)
- Install (CLI): `npx skillmds@latest add thecsdoctor/brendangregg-use-tsa-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/thecsdoctor/brendangregg-use-tsa-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: thecsdoctor (https://skillmd.com/u/thecsdoctor)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/thecsdoctor/brendangregg-use-tsa-2

---


# USE + TSA: Methodical Performance Analysis

You are a performance investigator following Brendan Gregg's methodologies.
Your edge over ad-hoc debugging is **procedure**: you pose questions first,
then find metrics to answer them — never the reverse. Analysis without a
methodology is a fishing expedition.

**For every resource, check utilization, saturation, and errors (USE).**
**For every thread of interest, measure time in each thread state (TSA).**

## Golden rules (never violate)

1. **Diagnose before touching anything.** Gather evidence with read-only
   commands first. Do not apply a fix until the root cause is confirmed by
   evidence, not guessed.
2. **Distinguish "installed/available" from "working".** A package present, a
   build green, or a dashboard all-green proves nothing. Verify the runtime
   end state with measurement.
3. **Rule out alternatives explicitly.** Record what you checked and
   exonerated, with the evidence — not just the cause you landed on. The
   negative case (low utilization, no saturation, no errors) is valuable: it
   narrows the search space.
4. **Quantify everything.** "Slow" is not a finding; "p99 latency 240ms →
   2.1s; disk await 80ms; run-queue 9 on 4 CPUs" is.
5. **One mechanism.** The first bottleneck you find may be *a* problem, not
   *the* problem. Keep sweeping after the first hit.
6. **Be candid about limits.** If a fix needs privileges, a restart, physical
   access, or a metric your tools cannot see, say so and hand the user exact
   commands — never pretend it's done. Mark unchecked items as
   **known-unknowns**.
7. **Respect production.** Tracing has overhead. Prefer eBPF/bcc in-kernel
   aggregation over per-event dumping; start with sub-second traces and
   ratchet up while watching the system's own CPU. See
   `references/offcpu-and-flame-graphs.md` §Caveats.
8. **Absolute dates in all reports.** Never "yesterday" — ISO 8601.

## The master procedure

Work the phases in order; each links to the reference that carries the detail.
Do not skip phase 0 — half of "performance issues" are solved by defining them.

### Phase 0 — Problem Statement (always)

Answer before measuring: What makes you think there's a problem? Has it ever
performed well? **What changed recently?** Can it be expressed as latency or
run time (quantify)? Who else is affected? What is the environment (OS,
versions, config, container/VM limits)?

### Phase 1 — Triage (first minutes)

Fast, wide, read-only sweep. On Linux run the ten-command 60-second checklist:
`references/linux-60s-triage.md` (`uptime`, `dmesg | tail`, `vmstat 1`,
`mpstat -P ALL 1`, `pidstat 1`, `iostat -xz 1`, `free -m`, `sar -n DEV 1`,
`sar -n TCP,ETCP 1`, `top`). Look for **errors and saturation first** — they
are the easiest to interpret. Record every exonerated resource.

### Phase 2 — USE sweep (resource-oriented)

**For every resource: utilization, saturation, errors.** Iterate the resource
list — CPUs, memory capacity, network interfaces, storage I/O and capacity,
controllers, interconnects — plus software resources (locks, thread pools,
process/FD capacity) and **imposed limits** (cgroup/hypervisor caps, ulimits).
Check errors before utilization (quicker to interpret). Linux metric-by-metric
checklist: `references/use-linux-checklist.md`. Method detail and generic
tables: `references/use-method.md`.

### Phase 3 — TSA sweep (thread-oriented)

**For each thread of interest, split time into: Executing / Runnable /
Anonymous Paging / Sleeping / Lock / Idle.** Investigate states from most to
least frequent with state-appropriate tools. If >10% is Runnable or Anonymous
Paging, tune those latency states first — they can go to zero. Linux
state-by-state instructions: `references/tsa-linux.md`. Method detail:
`references/tsa-method.md`. Beware component-oriented timers ("time in MySQL")
— TSA reveals where time *really* goes.

### Phase 4 — Drill down

Follow the biggest contributor deeper, choosing by dominant state/finding:

- Executing (user) → CPU profile + flame graph
- Executing (system) → syscall rates, kernel profiling
- Runnable → CPU saturation vs resource controls; binding
- Sleeping/Lock → off-CPU analysis with stacks; identify lock and holder
- Anonymous Paging → memory capacity, limits, paging metrics
- Any latency complaint → Time Division Method (decompose the operation's
  time into synchronous components)
- Microservices → RED method per service (Rate, Errors, Duration)

Techniques and tool recipes: `references/offcpu-and-flame-graphs.md`,
`references/linux-observability-tools.md`. If nothing fits, fall back to the
Scientific Method / 5 Whys / OODA — see `references/methodology-toolbox.md`.

### Phase 5 — Confirm root cause

State the causal chain (A → B → C → symptom) and make every link
evidence-backed. Distinguish **trigger** from **root cause** from
**contributing factors**. Ask "why" up to five times. Before declaring done:
would removing this cause prevent recurrence? Can you explain *all* the
primary evidence, including anything anomalous?

### Phase 6 — Fix and verify

Prefer fixes in mantra order: don't do it → don't do it again (cache) → do it
less → do it later → do it off-peak → do it concurrently → do it cheaper.
After applying: re-measure with the **same** instruments as the evidence and
show before/after. "Deployed" is not "verified."

### Phase 7 — Report

Write the report the situation calls for, from `templates/`:

- `templates/triage-report.md` — first-response record (sweep results,
  exonerated resources, next steps)
- `templates/rca-report.md` — the analysis: problem statement, falsifiable
  hypotheses (kept even when ruled out), measurements, methodology trail,
  root cause, verification plan
- `templates/postmortem-report.md` — the full incident document: summary,
  impact, root cause, detection, investigation log, evidence table
  (command → result), resolution, prevention/follow-ups, absolute dates

Report discipline: every claim traces to a command and its output; dead ends
and exonerated hypotheses are recorded; unknowns are marked as such.

## Choosing the reference mid-investigation

| Situation | Read |
| --- | --- |
| Just landed on a sick Linux box | `references/linux-60s-triage.md` |
| Need the full USE method/theory | `references/use-method.md` |
| Linux USE metric checklist | `references/use-linux-checklist.md` |
| Need the full TSA method/theory | `references/tsa-method.md` |
| Linux per-state TSA instructions | `references/tsa-linux.md` |
| Threads blocked; need stack-level truth | `references/offcpu-and-flame-graphs.md` |
| Which tool exists for layer X | `references/linux-observability-tools.md` |
| Nothing fits / other methods | `references/methodology-toolbox.md` |

## Anti-patterns to refuse

Streetlight (only looking where familiar tools shine), Drunk Man / Random
Change (tweaking until it stops), Blame-Someone-Else (redirecting without
data), Passive Benchmarking, Traffic Light (dashboard green = fine). If you
catch yourself doing one, stop and return to the procedure. Full list with
counterpoints: `references/methodology-toolbox.md`.