How do I install the Cuda Graph Trace Compaction skill?

Run npx skillmds@latest add pytorch/cuda-graph-trace-compaction in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Cuda Graph Trace Compaction safe to use?

This skill has not completed SkillMD's automated safety review yet. Capability flags: executes scripts. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Cuda Graph Trace Compaction?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Cuda Graph Trace Compaction free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Cuda Graph Trace Compaction?

pytorch (@pytorch) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Cuda Graph Trace Compaction by pytorch

Name: Cuda Graph Trace Compaction
Rating: 4.666666666666667 (3 reviews)
Author: pytorch

Cuda Graph Trace Compaction

Make a CUDA-graph PyTorch profiler trace readable in Perfetto by packing replay streams into semantic lanes, and optionally merge per-rank pipeline-parallel traces into one aligned timeline with send/recv flow arrows. Use when a trace captured with CUDA graphs shows hundreds of one-kernel stream rows, or when the user wants to view several PP rank traces together.

pytorch 59fa76f 3 files · 82.0 KB Updated Sep 10, 2026

File contents

CUDA Graph Trace Compaction

CUDA graph replay gives each captured region its own stream, so a profiler trace opens in Perfetto as hundreds of near-empty rows. scripts/compact_cuda_graph_trace.py repacks the slices into a few semantic lanes (compute, NCCL all-gather, reduce-scatter, ..., memcpy, memset).

Visualization only. Slices keep their name, pid, ts, dur, and args -- only the display placement (tid/args.stream) changes. Hand the user the original for measurement and the compacted copy for reading.

The script is agent-maintained, not a torchtitan API -- edit it for the trace in front of you rather than working around it. Common tweaks: _classify_slice and GROUP_ORDER/GROUP_LABELS for new lane groupings, PP_ANNOTATION_PATTERN for a different annotation scheme. Keep the tests passing.

# Single trace -> <name>_compacted.json.gz beside the input
python3 .claude/skills/cuda_graph_trace_compaction/scripts/compact_cuda_graph_trace.py \
    outputs/<run>/profile_trace/iteration_10/rank0_trace.json.gz

# All rank*_trace.json[.gz] in a directory -> pp_traces_merged_compacted.json.gz
python3 .claude/skills/cuda_graph_trace_compaction/scripts/compact_cuda_graph_trace.py \
    outputs/<run>/profile_trace/iteration_10 --merge-pp-ranks

-o sets the output path, --overwrite replaces an existing one.

Single-rank compaction includes rank-local PP arrows (UNSHARD -> compute, RECV -> compute, and compute -> SEND). Cross-rank SEND -> RECV arrows are added only when merging PP ranks.

Merging aligns ranks on a common baseTimeNanoseconds, stacks them in pipeline order, and draws flow arrows between the PP:<stage><op><microbatch> annotations that torch.distributed.pipelining emits (SEND_F -> RECV_F across ranks, RECV_F -> F -> SEND_F within one). It also connects each PP:<stage>UNSHARD annotation on the first two all-gather lanes to the first subsequent same-stage compute annotation on the first compute lane. No annotations means no arrows, but the merge still works.

The script prints a JSON summary and validates itself, raising if a measured slice changed or a lane ended up with overlapping slices. Worth a look: original_streams vs compacted_lanes for the compaction ratio, and pp_send_recv_unmatched (nonzero usually means a truncated capture window, not a pipeline bug).

Rank order comes from a torchtitan_real_pp group or a mesh_pp group description in distributedInfo.pg_config; absent that it sorts by global rank, which is only correct when PP rank order matches global rank order.

Tests

pytest .claude/skills/cuda_graph_trace_compaction/scripts/ -- hand-built traces, no GPU. Outside testpaths, so pytest tests/ skips it. Run it after touching the script; a broken invariant yields a plausible-looking wrong trace rather than an error.

---name: cuda-graph-trace-compactiondescription: Make a CUDA-graph PyTorch profiler trace readable in Perfetto by packing replay streams into semantic lanes, and optionally merge per-rank pipeline-parallel traces into one aligned timeline with send/recv flow arrows. Use when a trace captured with CUDA graphs shows hundreds of one-kernel stream rows, or when the user wants to view several PP rank traces together.---# CUDA Graph Trace CompactionCUDA graph replay gives each captured region its own stream, so a profilertrace opens in Perfetto as hundreds of near-empty rows.`scripts/compact_cuda_graph_trace.py` repacks the slices into a few semanticlanes (compute, NCCL all-gather, reduce-scatter, ..., memcpy, memset).**Visualization only.** Slices keep their name, `pid`, `ts`, `dur`, and `args`-- only the display placement (`tid`/`args.stream`) changes. Hand the user theoriginal for measurement and the compacted copy for reading.The script is agent-maintained, not a torchtitan API -- edit it for the tracein front of you rather than working around it. Common tweaks: `_classify_slice`and `GROUP_ORDER`/`GROUP_LABELS` for new lane groupings, `PP_ANNOTATION_PATTERN`for a different annotation scheme. Keep the tests passing.```bash# Single trace -> <name>_compacted.json.gz beside the inputpython3 .claude/skills/cuda_graph_trace_compaction/scripts/compact_cuda_graph_trace.py \ outputs/<run>/profile_trace/iteration_10/rank0_trace.json.gz# All rank*_trace.json[.gz] in a directory -> pp_traces_merged_compacted.json.gzpython3 .claude/skills/cuda_graph_trace_compaction/scripts/compact_cuda_graph_trace.py \ outputs/<run>/profile_trace/iteration_10 --merge-pp-ranks````-o` sets the output path, `--overwrite` replaces an existing one.Single-rank compaction includes rank-local PP arrows (`UNSHARD -> compute`,`RECV -> compute`, and `compute -> SEND`). Cross-rank `SEND -> RECV` arrowsare added only when merging PP ranks.Merging aligns ranks on a common `baseTimeNanoseconds`, stacks them inpipeline order, and draws flow arrows between the`PP:<stage><op><microbatch>` annotations that `torch.distributed.pipelining`emits (`SEND_F -> RECV_F` across ranks, `RECV_F -> F -> SEND_F` within one).It also connects each `PP:<stage>UNSHARD` annotation on the first twoall-gather lanes to the first subsequent same-stage compute annotation on thefirst compute lane.No annotations means no arrows, but the merge still works.The script prints a JSON summary and validates itself, raising if a measuredslice changed or a lane ended up with overlapping slices. Worth a look:`original_streams` vs `compacted_lanes` for the compaction ratio, and`pp_send_recv_unmatched` (nonzero usually means a truncated capture window,not a pipeline bug).Rank order comes from a `torchtitan_real_pp` group or a `mesh_pp` groupdescription in `distributedInfo.pg_config`; absent that it sorts by globalrank, which is only correct when PP rank order matches global rank order.## Tests`pytest .claude/skills/cuda_graph_trace_compaction/scripts/` -- hand-builttraces, no GPU. Outside `testpaths`, so `pytest tests/` skips it. Run it aftertouching the script; a broken invariant yields a plausible-looking wrong tracerather than an error.

Frequently asked questions