torch-perf-analysis: end-to-end PyTorch performance analysis
Skill Directory
${SKILL_DIR} refers to the directory containing this SKILL.md.
Overview
This skill is for analyzing a PyTorch workflow (training or inference) captured under nsys.
A companion skill, nsys-profile-analysis, covers veloq usage and generic NSys timeline patterns (iter regression, sync stalls, CUDA-graph drilldown, hotspot, bandwidth — see its references/cookbook.md). Consult it for veloq mechanics and general NSys workflows; this skill stays focused on PyTorch-specific recipes.
veloq is bundled inside talos — once you pip install talos, the binary is on PATH. Confirm with veloq --version.
Prerequisites
The user captures the .nsys-rep before invoking this skill (the skill does not capture it). Two steps:
Instrument the loop — no model argument; a process-global profiler hooks every nn.Module. Call enable() once up front, bracket the window with start()/stop(), and mark each step with step():
from talos.torch_perf_analysis import talos_profiler, nvtx_range
talos_profiler.enable() # optional: pre-load the observer up front
for step, batch in enumerate(loader):
if step == 50: talos_profiler.start() # open capture window
if step == 60: talos_profiler.stop()
talos_profiler.step() # mark a step boundary
loss = loss_fn(model(batch)); loss.backward()
optimizer.step(); optimizer.zero_grad()
Every outermost dispatched op that launches device work gets tagged with its module path + user call site (cwd-relative path:line); each kernel is later attributed to the line that launched it (forward and backward alike). Emission is a C++ RecordFunction observer (talos_observer.cpp, JIT-compiled on first use, ~1 min then cached; pre-build at install time with python -m talos.compile_observer) registered process-globally at the ATen dispatcher — it fires on every thread (framework workers and the C++ autograd threads), and measured overhead is ~4% of step time without nsys / ~5% marginal on top of an active nsys capture. enable() just pre-loads the extension so the mid-training start() stays clean. Wrap non-nn.Module code (an optimizer, a free helper) in nvtx_range("name", "Class") to give it its own attribution. Gated by TORCH_PERF_ANALYSIS_ENABLE — unset, every call is a no-op, so they can stay in the training script permanently. (Mechanics: ${SKILL_DIR}/python/profiler.py.)
Run under nsys — --capture-range=cudaProfilerApi records only the start()→stop() window. Use the minimal-overhead flag set (CPU sampling and context-switch tracing off, trace only cuda,nvtx — no cudnn/cublas/osrt):
TORCH_PERF_ANALYSIS_ENABLE=1 \
nsys profile --sample none --cpuctxsw=none \
--trace=cuda,nvtx \
--capture-range=cudaProfilerApi --capture-range-end=stop-shutdown \
-f true -o profile_output python3 <train_script>.py
(--capture-range-end=stop-shutdown terminates the process after the window; use stop + an external kill if the training loop must keep running.)
The user passes the resulting .nsys-rep via $ARGUMENTS.
Workflow
Two phases. Phase 1 is a fixed stats.py invocation that produces a workspace of structured JSON evidence; Phase 2 is the agent reading that workspace and producing the diagnosis (see the analysis directions in Step 2).
Workspace location is set with --workspace (default: .torch_perf_analysis in cwd). When running inside a torch-optimize loop, point it at that loop's folder (--workspace <root>/.torch_optimize/loop_<n>) so the evidence lands beside the round folders.
Step 1: Stats
TRACE="$ARGUMENTS"
python3 -m talos.torch_perf_analysis.python.stats "$TRACE"
The script:
- Runs
veloq summary "$TRACE" to materialize the parquet cache at <trace>.veloq/parquetdir/.
- Synthesizes backward attribution rows into
NVTX_EVENTS.parquet: for each backward-node event (XxxBackward0, seq carried in the NVTX int64 payload; legacy XxxBackward, seq = N text is also recognized), it looks up the forward event with the same seq (the talos:: range's own payload or a nested talos_seq: mark) and the talos::<func>#<path>@<Class>!!<file>:<line> range that enclosed it, and stamps that label verbatim over the enclosing autograd::engine::evaluate_function range — the full backward of that one forward op, which also covers the gradient-accumulate ops (aten::add …) sitting outside XxxBackward. (Placed strictly inside the evaluate_function, so analysis doesn't mistake it for the loss.backward() driver.) So a backward kernel keeps the forward's module + source line; direction is read from the surrounding autograd:: range, not the label. Idempotent.
- Invalidates veloq's NVTX-derived caches (
meta.bin, nvtx-tree.parquet, nvtx-parent.parquet) so subsequent queries pick up the synthesized rows. The CUPTI correlation cache (correlation.bin) stays — it doesn't depend on NVTX_EVENTS.
- Runs veloq queries for the run-level + movement evidence (
summary, hardware, slices for steps, stats --group-by nvtx-path for memcpy/sync, gaps for idle), and builds the per-stream GPU timeline directly from the parquet (correlate.build_timeline — kernel→NVTX launch correlation). Writes the per-step JSON artifacts above.
(A rendered perf_analysis_report.md is not emitted for now — the agent reads the JSON directly.)
Progress lines go to stderr ([stats] 1/4 ... through [stats] 4/4 ...).
Workspace artifacts produced
All JSON outputs and veloq's working caches (parquet + correlation index) land inside .torch_perf_analysis/. The original .nsys-rep is left untouched; stats.py symlinks it into the workspace so veloq's auto-generated <trace>.veloq/ directory sits there too. (See Step 2 for what each file means and how to use it.)
.torch_perf_analysis/
├── <trace_basename>.nsys-rep symlink to the user's trace
├── <trace_basename>.nsys-rep.veloq/
│ ├── parquetdir/ nsys-exported per-table parquet (includes the
│ │ augmented NVTX_EVENTS.parquet)
│ ├── meta.bin veloq's NVTX nesting / capabilities cache
│ ├── nvtx-tree.parquet
│ └── nvtx-parent.parquet
├── basics.json run-level basics, four keys:
│ trace_meta — trace_span_ns, capabilities
│ hardware — GPU / CPU / NIC inventory
│ steps — step_* slices + uniformity (CV)
│ typical_step — which step was picked (median) + reason
├── step_<N>/
│ ├── stream_<id>/ one folder per CUDA stream (compute / comm / copy):
│ │ ├── timeline_bucket.json this stream in 100 time segments (full detail)
│ │ ├── util.json just the 100 per-segment util values (quick scan)
│ │ └── source_attribution.json whole-stream GPU time by source line (desc)
│ ├── cpu_thread_<i>/ CPU side — one folder per thread:
│ │ └── torch_function_util.json torch-function util per segment + role (fwd/bwd)
│ ├── gpu_util.json device-level util: union across ALL streams,
│ │ bucketed + per-stream means — the aggregate
│ │ the per-stream util.json files can't show
│ ├── memcpy.json by_path: nvtx-path + source-line attribution
│ ├── sync.json by_path: hidden .item()/.cpu() stalls by source
│ ├── idle.json idle gaps > 100µs within the step
│ ├── exposed_idle.json hotspots ranked by step-time impact — each row
│ │ carries both levers (exposed_idle_ms + fusible_ms
│ │ + lever); by_region / by_source / by_sync_source
│ └── fusion_candidates.json source lines ranked by fusible (memory-bound
│ elementwise/reduction) GPU time — tech4 targets
(perf_analysis_report.md is not emitted for now — analyze the JSON directly.)
Capability gating
basics.json → trace_meta.capabilities.has_nvtx == false → user didn't run talos_profiler + --trace=nvtx,cuda. Tell them to recapture.
trace_meta.capabilities.has_cuda_contexts == false → kernel-to-NVTX attribution is unavailable; by_source lines fall back to (unattributed). Ask the user to recapture with full CUDA tracing.
Step 2: Analyze
stats.py produces a JSON workspace (no markdown report for now — read the JSON). What you get and what it means:
basics.json — trace span + capabilities, hardware, per-step durations + uniformity (CV), and which step was picked as typical (warmup / outlier steps are skipped). Sanity-check here first: very few steps, high CV, or a missing capability bit weakens everything downstream.
step_<N>/stream_<id>/ — one folder per CUDA stream (compute / comm / copy):
util.json — the stream's per-segment GPU util + min/mean/max. The quick scan. util is keyed by bucket index so a low value names its bucket directly: {"stream", "bucket_ns", "start_ns", "mean_util", "util": {"0": <util>, "1": <util>, …, "99": <util>}} — the key matches the bucket i in timeline_bucket.json.
timeline_bucket.json — full per-segment detail (clipped so util ≤ 100% even when one kernel spans several segments):{"i": 10, "start_ns": ..., "end_ns": ...,
"kernel_count": 25, "h2d_count": 0, "d2h_count": 0, "d2d_count": 0, "memset_count": 3,
"busy_ms": 11.26, "kernel_ms": 11.256, "memcpy_ms": 0.0, "idle_ms": 0.05, "util": 0.9956,
"by_source": [{"source": "<file>:<line>", "gpu_ms": <ms>, "count": <n>}, ...]}
by_source attributes the segment's GPU time to the source line (file:line) that launched it (forward and backward alike; (unattributed) = launched outside any talos:: range, e.g. NCCL internals).
source_attribution.json — the same attribution rolled up over the whole stream, largest first: [{"source": "<file>:<line>", "gpu_ms": <ms>, "count": <n>, "pct_of_step": <pct>}, ...].
step_<N>/cpu_thread_<i>/torch_function_util.json — CPU side, one folder per active thread (mirrors stream_<id>/): that thread's torch-function utilisation per segment (util keyed by bucket, same shape as a stream's util.json), plus role, cover_ms (torch-function time) and cuda_api_cover_ms (CUDA-API time).
This is a C++ / dispatch-utilisation metric: the fraction of the thread's wall time spent inside talos:: ranges — i.e. inside a torch.* call, from the moment it crosses from Python into the C++ dispatcher. Read it two ways:
- The gaps (
1 − util) are Python overhead. Time on the thread outside any talos:: range is the Python-side glue between torch ops — the interpreter, building arg/kwarg lists, control flow, pure-Python helpers. Wide gaps ⇒ the thread is spending its time in Python, not getting work to the GPU.
- The covered time (
util) is itself host-side C++ cost, not GPU compute. A talos:: range spans dispatch (operator resolution / kernel selection) plus the autograd graph-building (recording the grad_fn / next_edges) plus the kernel launch — all CPU work that merely issues GPU work. So even at high util this is launch/build cost on the host; it tells you the thread is dispatch-bound (busy feeding the GPU), which is only good insofar as the GPU stays busy (see Principle).
So high util ⇒ ops issued back-to-back, little Python glue (host cost is the unavoidable C++ dispatch/build); low util ⇒ the gaps dominate — Python between ops on a forward thread, the C++ autograd engine between ops on the backward worker (its ranges are the synthesized backward ranges). A thread with role: other runs no tracked torch ops (so util ≈ 0) but is busy launching raw CUDA — comm / copy / CUDA-graph (see cuda_api_cover_ms); it's listed because that activity is significant. Near-idle helper threads are filtered out. Note: a forward thread reads ~0 through the backward window because it's blocked there (not overhead) — cross-ref the backward thread, which is active then.
step_<N>/gpu_util.json — device-level utilisation: the union of busy time (kernel/memcpy/memset) across all streams, in the same 100-bucket format as a stream's util.json, plus per_stream_mean_util. Read this before the per-stream files: streams can each look moderate while the device is nearly saturated (busy windows interleave) — compare sum(per_stream_mean_util) against gpu.util: a large gap means heavy stream overlap (don't double-count "idle" that another stream fills); ≈ equal means no overlap, so a low-util bucket here is genuinely dead time. Totals match exposed_idle.json's gpu block.
step_<N>/memcpy.json / sync.json / idle.json — data movement, hidden host-sync stalls (.item()/.cpu()), and idle gaps, each with by_path carrying a source (file:line) + total_ms / pct_of_step. These are self-time views (how long the op itself ran / waited).
step_<N>/exposed_idle.json — the step-time-impact view: lines / regions ranked by how much they slow the step. Its core is the GPU idle each is responsible for. Device idle (window − union of kernel/memcpy/memset) is charged to the launch that ends each gap, and the host-sync-exposed slice is split out. Three cuts: by_region (NVTX-marker rollup, so one logical unit isn't split across lines), by_source (file:line + op + launch_count, for the Optimizer to act on), and by_sync_source (the slice actually caused by .item()/.cpu() waits). idle_split gives the headline host_sync_ms vs launch_bound_ms. Each by_region / by_source row also carries the busy lever — busy_ms and fusible_ms (memory-bound time reducible by fusion) — plus a per-row lever hint, so the lever is chosen per hotspot (a model mixes idle-bound regions like the optimizer / masking and busy-bound ones like embedding / layernorm). A big self-time .item() whose host wait overlaps GPU-busy work contributes ~0 exposed idle — removing it moves nothing. Caveats: the "next launch" attribution folds the Python glue before a launch into it (a reasonable "what is the GPU waiting for", not exact causation); fusible_ms counts only fusible kinds, so a compute-bound region (large busy_ms, small fusible_ms) correctly stays on the idle lever.
step_<N>/fusion_candidates.json — source-line regions ranked by fusible GPU time (the kernel-fusion / tech4 angle). Each kernel is classified by its demangled name; only memory-bound families count toward fusible_gpu_ms: elementwise (pointwise ops — nan_to_num, clamp, where, activations…, fuses directly) and reduction (sum / mean / norm / softmax — fuses with its surrounding pointwise, not in isolation). Compute-bound (gemm / conv / attention) and non-fusible (comm / sort) kernels are shown in by_kind but excluded. CUPTI has no per-kernel byte count, so gpu_ms is the memory-traffic proxy: a region ranks high whether it is launch-bound (many tiny kernels, low util) or bandwidth-bound (few large kernels at high util — invisible to the util timeline). Each row: {source, fusible_gpu_ms, fusible_kernel_count, total_gpu_ms, total_kernel_count, pct_of_step, by_kind}. This is the busy-lever detail: exposed_idle.json already flags a hotspot as busy via its fusible_ms; come here for the per-kind breakdown of what is actually fusible.
Principle
The analysis is GPU-centric. The goal is keeping the GPU busy, so findings are anchored in GPU under-utilization, stalls, or work on the critical path. The per-stream util timeline is the source of truth.
CPU-side costs are relevant when they block GPU progress. Cross-reference the same time window on the compute stream's util.json / timeline_bucket.json: low utilization identifies a real stall, while sustained utilization indicates overlapped work. Each finding should tie back to the GPU idle or critical-path work it explains.
For launch-bound regions, report launch evidence: kernel count, CUDA runtime launch API count/time (veloq stats --type runtime --nvtx ...), and the enclosing CPU NVTX range or local CPU timer (veloq slices ...). Use GPU util, idle gaps, and step duration to judge whether that launch cost is on the critical path.
Analysis directions
There's no fixed sequence here — these are complementary angles, and you decide how to follow them. The general idea: first read the coarse, whole-step stats to build a mental picture of the timeline; then zoom into whatever looks off (a low-util stretch, a hot source line) and drill the root cause with veloq. The rest is yours to drive — follow the evidence.
Build the picture first. Pick the compute stream (highest busy_ms / kernel_count; comm is D2D/NCCL-heavy, copy is H2D/D2H-heavy). Then look from either angle:
- GPU utilization — start with
gpu_util.json (device level, union across all streams): its gpu.util is the single headline number, and a low-util bucket there is genuinely dead time (no stream is filling it). Then drill the compute stream's util.json for the same bucket window; the comm / copy streams' util.json at that index hints at why (comm busy while compute idles → exposed communication; nothing busy → a CPU/launch or host-sync gap).
- Rank hotspots on
exposed_idle.json (detailed under the artifacts above) — rank each by the larger of its exposed_idle_ms and fusible_ms, then follow its lever: idle → read idle_split (host_sync vs launch_bound); busy → drill fusion_candidates.json for the fusible by_kind breakdown. gpu.idle_pct is overall orientation only.
- CPU / dispatch utilisation — when the compute stream has a low-util stretch and nothing on comm/copy explains it, read
cpu_thread_<i>/torch_function_util.json over that bucket window. Low torch-function util there means the gap is Python overhead (the thread is between torch ops, not feeding the GPU); high util means the host is dispatch/launch-bound (busy in C++ issuing work). Either way the cost is host-side, so it only matters when it lines up with GPU idle — cross-ref the compute stream's util.json at the same bucket.
Then drill into the suspicious / hot regions. For a bucket i worth a closer look, open timeline_bucket.json at i (its by_source, counts, kernel_ms vs memcpy_ms) and query that start_ns/end_ns window with veloq to pin the root cause. Things the evidence often points at: a hidden host-sync stall (a big cudaStreamSynchronize from a .item() / .cpu() / host-side print line), a pageable HtoD copy, a line of many tiny kernels, exposed (non-overlapped) communication, or a gap between steps. memcpy.json / sync.json source is null when the event sits under a non-talos:: region — read its raw nvtx_path or veloq correlate <key>. (veloq mechanics: companion nsys-profile-analysis skill.)
Report each finding with the evidence that matches its bottleneck: GPU ms / % for compute or memory-traffic findings; launch count, runtime API time, and CPU-range time for launch-bound findings.
Drilling deeper with veloq
The stats phase extracts a fixed evidence set, sufficient for most diagnoses. For sharper queries, run veloq directly — after stats.py's parquet synthesis, NVTX_EVENTS carries talos:: ranges on both forward and backward ops (backward gets the forward label verbatim), so an nvtx-path glob matches a module in both directions:
- Per-step bucket analysis (100 bins, with per-kind breakdown):
START=...; END=...; BUCKET=$(((END - START) / 100))
veloq timeline "$TRACE" --from @$START --to @$END --bucket "${BUCKET}"ns \
| python3 -c 'import json,sys
for r in json.load(sys.stdin)["data"]["rows"]:
print(r["key"], r["busy_ns"] / r["total_ns"], r["breakdown"])'
- Cross-step regression diff: key both timeline outputs by
.data.rows[].key and compare the buckets that moved.
- Drilling into one class's kernels:
veloq stats T --type kernel --nvtx 'talos::*@Linear*' --group-by demangled --limit 10 (matches forward + backward; to isolate backward, intersect with the autograd::engine::evaluate_function range).
- CPU launch site for one event:
veloq inspect "$TRACE" <row_id> / veloq correlate "$TRACE" <row_id>.
For full veloq mechanics, defer to the companion nsys-profile-analysis skill.
Error handling
veloq: command not found → pip install --force-reinstall talos (the binary ships bundled with the wheel).
nsys: command not found on the first export call → install nsys ≥ 2024.6 or supply a pre-exported _pqtdir/ directly.
pyarrow required for parquet synthesis → pip install pyarrow (usually already present in PyTorch envs).
basics.json → trace_meta.capabilities.has_nvtx == false → user didn't run talos_profiler + --trace=nvtx,cuda.
trace_meta.capabilities.has_cuda_contexts == false → kernel-to-NVTX attribution unavailable; timeline by_source falls back to (unattributed). Ask user to recapture with full CUDA tracing.
basics.json → steps.count == 0 / typical_step == null → no step_* ranges; either the user didn't set TORCH_PERF_ANALYSIS_ENABLE=1 (most common — check stderr for "disabled; set TORCH_PERF_ANALYSIS_ENABLE=1"), or talos_profiler.start() / .stop() didn't bracket the captured steps, or the steps weren't marked with talos_profiler.step().
- timeline
by_source is all forward lines (no backward) → bwd synthesis didn't find any seq matches; check that seq-carrying events exist in the trace (talos:: ranges / talos_seq: marks with an int64 payload — emitted automatically between talos_profiler.start() and .stop()).
jq: command not found → only the nsys-profile-analysis / ncu-profile-analysis example commands use it. Install it (apt install jq / brew install jq) or read the JSON with python3 -c 'import json,sys; ...'.
- Deep veloq-specific issues → consult the
nsys-profile-analysis skill.
1---2name: torch-perf-analysis3description: "Analyze an existing nsys trace of a PyTorch workload (training or inference, NVTX-instrumented) and produce an end-to-end performance diagnosis. Uses the bundled `veloq` CLI for evidence extraction."4argument-hint: "<nsys_trace_file>"5metadata:6 source: https://skillmd.com/skills/nvidia/torch-perf-analysis7---89# torch-perf-analysis: end-to-end PyTorch performance analysis1011## Skill Directory1213`${SKILL_DIR}` refers to the directory containing this SKILL.md.1415## Overview1617This skill is for analyzing a PyTorch workflow (training or inference) captured under nsys.1819A companion skill, **`nsys-profile-analysis`**, covers veloq usage and generic NSys timeline patterns (iter regression, sync stalls, CUDA-graph drilldown, hotspot, bandwidth — see its `references/cookbook.md`). Consult it for veloq mechanics and general NSys workflows; this skill stays focused on PyTorch-specific recipes.2021`veloq` is bundled inside `talos` — once you `pip install talos`, the binary is on `PATH`. Confirm with `veloq --version`.2223---2425## Prerequisites2627The user captures the `.nsys-rep` before invoking this skill (the skill does **not** capture it). Two steps:28291. **Instrument the loop** — no model argument; a process-global profiler hooks every `nn.Module`. Call `enable()` once up front, bracket the window with `start()`/`stop()`, and mark each step with `step()`:3031 ```python32 from talos.torch_perf_analysis import talos_profiler, nvtx_range3334 talos_profiler.enable() # optional: pre-load the observer up front3536 for step, batch in enumerate(loader):37 if step == 50: talos_profiler.start() # open capture window38 if step == 60: talos_profiler.stop()39 talos_profiler.step() # mark a step boundary40 loss = loss_fn(model(batch)); loss.backward()41 optimizer.step(); optimizer.zero_grad()42 ```4344 Every outermost dispatched op that launches device work gets tagged with its module path + user call site (cwd-relative path:line); each kernel is later attributed to the line that launched it (forward and backward alike). Emission is a **C++ RecordFunction observer** (`talos_observer.cpp`, JIT-compiled on first use, ~1 min then cached; pre-build at install time with `python -m talos.compile_observer`) registered process-globally at the ATen dispatcher — it fires on *every* thread (framework workers and the C++ autograd threads), and measured overhead is ~4% of step time without nsys / ~5% marginal on top of an active nsys capture. `enable()` just pre-loads the extension so the mid-training `start()` stays clean. Wrap non-`nn.Module` code (an optimizer, a free helper) in `nvtx_range("name", "Class")` to give it its own attribution. Gated by `TORCH_PERF_ANALYSIS_ENABLE` — unset, every call is a no-op, so they can stay in the training script permanently. (Mechanics: `${SKILL_DIR}/python/profiler.py`.)45462. **Run under nsys** — `--capture-range=cudaProfilerApi` records only the `start()`→`stop()` window. Use the minimal-overhead flag set (CPU sampling and context-switch tracing off, trace only `cuda,nvtx` — no cudnn/cublas/osrt):4748 ```bash49 TORCH_PERF_ANALYSIS_ENABLE=1 \50 nsys profile --sample none --cpuctxsw=none \51 --trace=cuda,nvtx \52 --capture-range=cudaProfilerApi --capture-range-end=stop-shutdown \53 -f true -o profile_output python3 <train_script>.py54 ```5556 (`--capture-range-end=stop-shutdown` terminates the process after the window; use `stop` + an external kill if the training loop must keep running.)5758The user passes the resulting `.nsys-rep` via `$ARGUMENTS`.5960---6162## Workflow6364Two phases. Phase 1 is a fixed `stats.py` invocation that produces a workspace of structured JSON evidence; Phase 2 is the agent reading that workspace and producing the diagnosis (see the analysis directions in Step 2).6566Workspace location is set with `--workspace` (default: `.torch_perf_analysis` in cwd). When running inside a `torch-optimize` loop, point it at that loop's folder (`--workspace <root>/.torch_optimize/loop_<n>`) so the evidence lands beside the round folders.6768---6970## Step 1: Stats7172```bash73TRACE="$ARGUMENTS"74python3 -m talos.torch_perf_analysis.python.stats "$TRACE"75```7677The script:78791. Runs `veloq summary "$TRACE"` to materialize the parquet cache at `<trace>.veloq/parquetdir/`.802. **Synthesizes backward attribution rows** into `NVTX_EVENTS.parquet`: for each backward-node event (`XxxBackward0`, seq carried in the NVTX int64 payload; legacy `XxxBackward, seq = N` text is also recognized), it looks up the forward event with the same `seq` (the `talos::` range's own payload or a nested `talos_seq:` mark) and the `talos::<func>#<path>@<Class>!!<file>:<line>` range that enclosed it, and stamps that label *verbatim* over the **enclosing `autograd::engine::evaluate_function` range** — the full backward of that one forward op, which also covers the gradient-accumulate ops (`aten::add` …) sitting outside `XxxBackward`. (Placed strictly inside the `evaluate_function`, so analysis doesn't mistake it for the `loss.backward()` driver.) So a backward kernel keeps the forward's module + source line; direction is read from the surrounding `autograd::` range, not the label. Idempotent.813. Invalidates veloq's NVTX-derived caches (`meta.bin`, `nvtx-tree.parquet`, `nvtx-parent.parquet`) so subsequent queries pick up the synthesized rows. The CUPTI correlation cache (`correlation.bin`) stays — it doesn't depend on NVTX_EVENTS.824. Runs veloq queries for the run-level + movement evidence (`summary`, `hardware`, `slices` for steps, `stats --group-by nvtx-path` for memcpy/sync, `gaps` for idle), and builds the per-stream GPU timeline **directly from the parquet** (`correlate.build_timeline` — kernel→NVTX launch correlation). Writes the per-step JSON artifacts above.83 (A rendered `perf_analysis_report.md` is not emitted for now — the agent reads the JSON directly.)8485Progress lines go to stderr (`[stats] 1/4 ...` through `[stats] 4/4 ...`).8687### Workspace artifacts produced8889All JSON outputs **and** veloq's working caches (parquet + correlation index) land inside `.torch_perf_analysis/`. The original `.nsys-rep` is left untouched; stats.py symlinks it into the workspace so veloq's auto-generated `<trace>.veloq/` directory sits there too. (See Step 2 for what each file means and how to use it.)9091```92.torch_perf_analysis/93├── <trace_basename>.nsys-rep symlink to the user's trace94├── <trace_basename>.nsys-rep.veloq/95│ ├── parquetdir/ nsys-exported per-table parquet (includes the96│ │ augmented NVTX_EVENTS.parquet)97│ ├── meta.bin veloq's NVTX nesting / capabilities cache98│ ├── nvtx-tree.parquet99│ └── nvtx-parent.parquet100├── basics.json run-level basics, four keys:101│ trace_meta — trace_span_ns, capabilities102│ hardware — GPU / CPU / NIC inventory103│ steps — step_* slices + uniformity (CV)104│ typical_step — which step was picked (median) + reason105├── step_<N>/106│ ├── stream_<id>/ one folder per CUDA stream (compute / comm / copy):107│ │ ├── timeline_bucket.json this stream in 100 time segments (full detail)108│ │ ├── util.json just the 100 per-segment util values (quick scan)109│ │ └── source_attribution.json whole-stream GPU time by source line (desc)110│ ├── cpu_thread_<i>/ CPU side — one folder per thread:111│ │ └── torch_function_util.json torch-function util per segment + role (fwd/bwd)112│ ├── gpu_util.json device-level util: union across ALL streams,113│ │ bucketed + per-stream means — the aggregate114│ │ the per-stream util.json files can't show115│ ├── memcpy.json by_path: nvtx-path + source-line attribution116│ ├── sync.json by_path: hidden .item()/.cpu() stalls by source117│ ├── idle.json idle gaps > 100µs within the step118│ ├── exposed_idle.json hotspots ranked by step-time impact — each row119│ │ carries both levers (exposed_idle_ms + fusible_ms120│ │ + lever); by_region / by_source / by_sync_source121│ └── fusion_candidates.json source lines ranked by fusible (memory-bound122│ elementwise/reduction) GPU time — tech4 targets123(perf_analysis_report.md is not emitted for now — analyze the JSON directly.)124```125126### Capability gating127128- `basics.json` → `trace_meta.capabilities.has_nvtx == false` → user didn't run `talos_profiler` + `--trace=nvtx,cuda`. Tell them to recapture.129- `trace_meta.capabilities.has_cuda_contexts == false` → kernel-to-NVTX attribution is unavailable; `by_source` lines fall back to `(unattributed)`. Ask the user to recapture with full CUDA tracing.130131---132133## Step 2: Analyze134135`stats.py` produces a JSON workspace (no markdown report for now — read the JSON). What you get and what it means:136137- **`basics.json`** — trace span + capabilities, hardware, per-step durations + uniformity (CV), and which step was picked as *typical* (warmup / outlier steps are skipped). Sanity-check here first: very few steps, high CV, or a missing capability bit weakens everything downstream.138- **`step_<N>/stream_<id>/`** — one folder per CUDA stream (compute / comm / copy):139 - **`util.json`** — the stream's per-segment GPU util + min/mean/max. The quick scan. `util` is keyed by bucket index so a low value names its bucket directly: `{"stream", "bucket_ns", "start_ns", "mean_util", "util": {"0": <util>, "1": <util>, …, "99": <util>}}` — the key matches the bucket `i` in `timeline_bucket.json`.140 - **`timeline_bucket.json`** — full per-segment detail (clipped so `util` ≤ 100% even when one kernel spans several segments):141 ```json142 {"i": 10, "start_ns": ..., "end_ns": ...,143 "kernel_count": 25, "h2d_count": 0, "d2h_count": 0, "d2d_count": 0, "memset_count": 3,144 "busy_ms": 11.26, "kernel_ms": 11.256, "memcpy_ms": 0.0, "idle_ms": 0.05, "util": 0.9956,145 "by_source": [{"source": "<file>:<line>", "gpu_ms": <ms>, "count": <n>}, ...]}146 ```147 `by_source` attributes the segment's GPU time to the **source line** (`file:line`) that launched it (forward and backward alike; `(unattributed)` = launched outside any `talos::` range, e.g. NCCL internals).148 - **`source_attribution.json`** — the same attribution rolled up over the **whole stream**, largest first: `[{"source": "<file>:<line>", "gpu_ms": <ms>, "count": <n>, "pct_of_step": <pct>}, ...]`.149- **`step_<N>/cpu_thread_<i>/torch_function_util.json`** — CPU side, one folder per **active** thread (mirrors `stream_<id>/`): that thread's **torch-function utilisation** per segment (`util` keyed by bucket, same shape as a stream's `util.json`), plus `role`, `cover_ms` (torch-function time) and `cuda_api_cover_ms` (CUDA-API time).150151 This is a **C++ / dispatch-utilisation** metric: the fraction of the thread's wall time spent *inside* `talos::` ranges — i.e. inside a `torch.*` call, from the moment it crosses from Python into the C++ dispatcher. Read it two ways:152 1. **The gaps (`1 − util`) are Python overhead.** Time on the thread *outside* any `talos::` range is the Python-side glue between torch ops — the interpreter, building arg/kwarg lists, control flow, pure-Python helpers. Wide gaps ⇒ the thread is spending its time in Python, not getting work to the GPU.153 2. **The covered time (`util`) is itself host-side C++ cost, not GPU compute.** A `talos::` range spans dispatch (operator resolution / kernel selection) plus the autograd graph-building (recording the `grad_fn` / `next_edges`) plus the kernel launch — all CPU work that merely *issues* GPU work. So even at high util this is launch/build cost on the host; it tells you the thread is dispatch-bound (busy feeding the GPU), which is only good insofar as the GPU stays busy (see Principle).154155 So **high util ⇒ ops issued back-to-back, little Python glue** (host cost is the unavoidable C++ dispatch/build); **low util ⇒ the gaps dominate** — **Python** between ops on a `forward` thread, the **C++ autograd engine** between ops on the `backward` worker (its ranges are the synthesized backward ranges). A thread with **`role: other`** runs no tracked torch ops (so `util` ≈ 0) but is busy launching raw CUDA — comm / copy / CUDA-graph (see `cuda_api_cover_ms`); it's listed because that activity is significant. Near-idle helper threads are filtered out. Note: a forward thread reads ~0 through the backward window because it's *blocked* there (not overhead) — cross-ref the backward thread, which is active then.156- **`step_<N>/gpu_util.json`** — **device-level** utilisation: the union of busy time (kernel/memcpy/memset) across **all** streams, in the same 100-bucket format as a stream's `util.json`, plus `per_stream_mean_util`. Read this **before** the per-stream files: streams can each look moderate while the device is nearly saturated (busy windows interleave) — compare `sum(per_stream_mean_util)` against `gpu.util`: a large gap means heavy stream overlap (don't double-count "idle" that another stream fills); ≈ equal means no overlap, so a low-util bucket here is genuinely dead time. Totals match `exposed_idle.json`'s `gpu` block.157- **`step_<N>/memcpy.json` / `sync.json` / `idle.json`** — data movement, hidden host-sync stalls (`.item()`/`.cpu()`), and idle gaps, each with `by_path` carrying a **`source`** (`file:line`) + `total_ms` / `pct_of_step`. These are **self-time** views (how long the op itself ran / waited).158- **`step_<N>/exposed_idle.json`** — the **step-time-impact** view: lines / regions ranked by how much they slow the step. Its core is *the GPU idle each is responsible for*. Device idle (window − union of kernel/memcpy/memset) is charged to the launch that ends each gap, and the host-sync-exposed slice is split out. Three cuts: `by_region` (NVTX-marker rollup, so one logical unit isn't split across lines), `by_source` (`file:line` + `op` + `launch_count`, for the Optimizer to act on), and `by_sync_source` (the slice actually caused by `.item()`/`.cpu()` waits). `idle_split` gives the headline `host_sync_ms` vs `launch_bound_ms`. Each `by_region` / `by_source` row **also carries the busy lever** — `busy_ms` and `fusible_ms` (memory-bound time reducible by fusion) — plus a per-row `lever` hint, so the lever is chosen **per hotspot** (a model mixes idle-bound regions like the optimizer / masking and busy-bound ones like embedding / layernorm). A big self-time `.item()` whose host wait overlaps GPU-busy work contributes ~0 exposed idle — removing it moves nothing. Caveats: the "next launch" attribution folds the Python glue *before* a launch into it (a reasonable "what is the GPU waiting for", not exact causation); `fusible_ms` counts only fusible kinds, so a compute-bound region (large `busy_ms`, small `fusible_ms`) correctly stays on the idle lever.159- **`step_<N>/fusion_candidates.json`** — source-line regions ranked by **fusible GPU time** (the kernel-fusion / tech4 angle). Each kernel is classified by its demangled name; only memory-bound families count toward `fusible_gpu_ms`: **`elementwise`** (pointwise ops — `nan_to_num`, `clamp`, `where`, activations…, fuses directly) and **`reduction`** (`sum` / `mean` / `norm` / `softmax` — fuses *with its surrounding pointwise*, not in isolation). Compute-bound (`gemm` / `conv` / `attention`) and non-fusible (`comm` / `sort`) kernels are shown in `by_kind` but excluded. CUPTI has no per-kernel byte count, so `gpu_ms` is the memory-traffic proxy: a region ranks high whether it is **launch-bound** (many tiny kernels, low util) or **bandwidth-bound** (few large kernels at high util — invisible to the util timeline). Each row: `{source, fusible_gpu_ms, fusible_kernel_count, total_gpu_ms, total_kernel_count, pct_of_step, by_kind}`. This is the **busy-lever detail**: `exposed_idle.json` already flags a hotspot as `busy` via its `fusible_ms`; come here for the per-kind breakdown of what is actually fusible.160161### Principle162163The analysis is **GPU-centric**. The goal is keeping the GPU busy, so findings are anchored in GPU under-utilization, stalls, or work on the critical path. The per-stream `util` timeline is the source of truth.164165CPU-side costs are relevant when they block GPU progress. Cross-reference the same time window on the compute stream's `util.json` / `timeline_bucket.json`: low utilization identifies a real stall, while sustained utilization indicates overlapped work. Each finding should tie back to the GPU idle or critical-path work it explains.166167For launch-bound regions, report launch evidence: kernel count, CUDA runtime launch API count/time (`veloq stats --type runtime --nvtx ...`), and the enclosing CPU NVTX range or local CPU timer (`veloq slices ...`). Use GPU util, idle gaps, and step duration to judge whether that launch cost is on the critical path.168169### Analysis directions170171There's no fixed sequence here — these are complementary angles, and you decide how to follow them. The general idea: first read the coarse, whole-step stats to build a mental picture of the timeline; then zoom into whatever looks off (a low-util stretch, a hot source line) and drill the root cause with `veloq`. The rest is yours to drive — follow the evidence.172173**Build the picture first.** Pick the **compute stream** (highest busy_ms / kernel_count; comm is D2D/NCCL-heavy, copy is H2D/D2H-heavy). Then look from either angle:174- **GPU utilization** — start with `gpu_util.json` (device level, union across all streams): its `gpu.util` is the single headline number, and a low-util bucket there is *genuinely* dead time (no stream is filling it). Then drill the compute stream's `util.json` for the same bucket window; the comm / copy streams' `util.json` at that index hints at why (comm busy while compute idles → exposed communication; nothing busy → a CPU/launch or host-sync gap).175- **Rank hotspots on `exposed_idle.json`** (detailed under the artifacts above) — rank each by the larger of its `exposed_idle_ms` and `fusible_ms`, then follow its `lever`: `idle` → read `idle_split` (`host_sync` vs `launch_bound`); `busy` → drill `fusion_candidates.json` for the fusible `by_kind` breakdown. `gpu.idle_pct` is overall orientation only.176- **CPU / dispatch utilisation** — when the compute stream has a low-util stretch and nothing on comm/copy explains it, read `cpu_thread_<i>/torch_function_util.json` over that bucket window. Low torch-function util there means the gap is **Python overhead** (the thread is between torch ops, not feeding the GPU); high util means the host is **dispatch/launch-bound** (busy in C++ issuing work). Either way the cost is host-side, so it only matters when it lines up with GPU idle — cross-ref the compute stream's `util.json` at the same bucket.177178**Then drill into the suspicious / hot regions.** For a bucket `i` worth a closer look, open `timeline_bucket.json` at `i` (its `by_source`, counts, `kernel_ms` vs `memcpy_ms`) and query that `start_ns`/`end_ns` window with `veloq` to pin the root cause. Things the evidence often points at: a hidden host-sync stall (a big `cudaStreamSynchronize` from a `.item()` / `.cpu()` / host-side print line), a pageable `HtoD` copy, a line of many tiny kernels, exposed (non-overlapped) communication, or a gap between steps. `memcpy.json` / `sync.json` `source` is null when the event sits under a non-`talos::` region — read its raw `nvtx_path` or `veloq correlate <key>`. (veloq mechanics: companion `nsys-profile-analysis` skill.)179180Report each finding with the evidence that matches its bottleneck: GPU ms / % for compute or memory-traffic findings; launch count, runtime API time, and CPU-range time for launch-bound findings.181182---183184## Drilling deeper with veloq185186The stats phase extracts a fixed evidence set, sufficient for most diagnoses. For sharper queries, run `veloq` directly — after stats.py's parquet synthesis, NVTX_EVENTS carries `talos::` ranges on both forward and backward ops (backward gets the forward label verbatim), so an nvtx-path glob matches a module in both directions:187188- **Per-step bucket analysis** (100 bins, with per-kind breakdown):189 ```bash190 START=...; END=...; BUCKET=$(((END - START) / 100))191 veloq timeline "$TRACE" --from @$START --to @$END --bucket "${BUCKET}"ns \192 | python3 -c 'import json,sys193for r in json.load(sys.stdin)["data"]["rows"]:194 print(r["key"], r["busy_ns"] / r["total_ns"], r["breakdown"])'195 ```196- **Cross-step regression diff**: key both timeline outputs by `.data.rows[].key` and compare the buckets that moved.197- **Drilling into one class's kernels**: `veloq stats T --type kernel --nvtx 'talos::*@Linear*' --group-by demangled --limit 10` (matches forward + backward; to isolate backward, intersect with the `autograd::engine::evaluate_function` range).198- **CPU launch site for one event**: `veloq inspect "$TRACE" <row_id>` / `veloq correlate "$TRACE" <row_id>`.199200For full veloq mechanics, defer to the companion `nsys-profile-analysis` skill.201202---203204## Error handling205206- `veloq: command not found` → `pip install --force-reinstall talos` (the binary ships bundled with the wheel).207- `nsys: command not found` on the first export call → install `nsys ≥ 2024.6` or supply a pre-exported `_pqtdir/` directly.208- `pyarrow required for parquet synthesis` → `pip install pyarrow` (usually already present in PyTorch envs).209- `basics.json` → `trace_meta.capabilities.has_nvtx == false` → user didn't run `talos_profiler` + `--trace=nvtx,cuda`.210- `trace_meta.capabilities.has_cuda_contexts == false` → kernel-to-NVTX attribution unavailable; timeline `by_source` falls back to `(unattributed)`. Ask user to recapture with full CUDA tracing.211- `basics.json` → `steps.count == 0` / `typical_step == null` → no `step_*` ranges; either the user didn't set `TORCH_PERF_ANALYSIS_ENABLE=1` (most common — check stderr for "disabled; set TORCH_PERF_ANALYSIS_ENABLE=1"), or `talos_profiler.start() / .stop()` didn't bracket the captured steps, or the steps weren't marked with `talos_profiler.step()`.212- timeline `by_source` is all forward lines (no backward) → bwd synthesis didn't find any seq matches; check that seq-carrying events exist in the trace (`talos::` ranges / `talos_seq:` marks with an int64 payload — emitted automatically between `talos_profiler.start()` and `.stop()`).213- `jq: command not found` → only the `nsys-profile-analysis` / `ncu-profile-analysis` example commands use it. Install it (`apt install jq` / `brew install jq`) or read the JSON with `python3 -c 'import json,sys; ...'`.214- Deep veloq-specific issues → consult the `nsys-profile-analysis` skill.