CUDA Graphs for PyTorch
CUDA Graphs capture a sequence of GPU operations once and replay them with
minimal CPU overhead. This skill guides applying CUDA Graphs to PyTorch
training and inference workloads using native PyTorch APIs, Transformer
Engine, and Megatron-LM.
When to Use
Reach for this skill when you encounter:
- Triggers: User wants to optimize with CUDA Graphs, reduce kernel launch
overhead, or speed up training/inference loops
- Symptoms: Low GPU utilization (<80%), many small kernel launches (<50 us
each), CPU-bound training, high kernel launch latency visible in Nsight
Systems profiles
- Keywords: "CUDA graph", "torch.cuda.graph", "make_graphed_callables",
"reduce-overhead", "graph capture", "graph replay", "kernel launch overhead",
"CudaGraphManager", "FullCudaGraphWrapper", "full-iteration graph", "stream
capture"
Do NOT use this skill for:
- General PyTorch performance tuning unrelated to kernel launch overhead
- CUDA kernel development or custom CUDA C++ code
- Host-device sync elimination only (use perf-torch-sync-free skill instead)
- Nsight Systems profiling (use perf-nsight-systems skill)
- TensorFlow/JAX graph compilation (different APIs entirely)
Requirements
| Dependency |
Version |
Notes |
| PyTorch |
>= 1.10 |
torch.cuda.graph() available |
| CUDA |
>= 11.0 |
Graph update APIs |
| GPU |
NVIDIA (any) |
Required for CUDA |
| Nsight Systems |
any |
Optional, for profiling |
| APEX |
any |
Optional, for capturable optimizers |
| Transformer Engine |
>= 2.2 |
Optional, for FP8-aware graphing |
| Megatron-LM |
core >= 0.14.0 |
Optional, for CudaGraphManager / FullCudaGraphWrapper |
API Selection Guide
Choose the API based on your framework and performance needs.
| Situation |
API |
Workflow |
| Quick experiment, unknown graph boundaries |
torch.compile(mode="reduce-overhead") |
Workflow 2 |
| Training, need autograd, no FP8/PP |
torch.cuda.make_graphed_callables() |
Workflow 3 |
| Any PyTorch model, FP8 or PP support |
TE make_graphed_callables |
Workflow 4 |
| Megatron-LM, per-layer, automatic |
MCore CudaGraphManager |
Workflow 5 |
| Maximum perf, full-iteration capture |
MCore FullCudaGraphWrapper |
Workflow 6 |
| Full manual control, custom pipelines |
torch.cuda.graph() |
Workflow 7 |
Decision flowchart:
- Using Megatron-LM with FP8/PP?
- Yes, want maximum perf with static workload --> Workflow 6 (FullCudaGraphWrapper)
- Yes, want per-layer automatic graphing --> Workflow 5 (CudaGraphManager)
- Yes, want manual control over what gets graphed --> Workflow 4 (TE make_graphed_callables)
- Using Transformer Engine without Megatron?
- Yes, need FP8 or PP --> Workflow 4 (TE make_graphed_callables)
- General PyTorch?
- Want zero effort, okay with fragmented graphs --> Workflow 2 (torch.compile)
- Want autograd support, training loop --> Workflow 3 (PyTorch make_graphed_callables)
- Want full manual control --> Workflow 7 (torch.cuda.graph)
Strategy: Start with the highest-level API available for your framework.
Move to lower-level APIs only if you need more control, hit limitations, or
do not achieve the expected performance improvement.
Workflows
Workflow 1: Profile and Decide Whether Graphs Help
Goal: Determine if CUDA Graphs will benefit your workload before investing
effort.
- Profile with Nsight Systems:
nsys profile --cuda-graph-trace=graph python train.py
- Check GPU utilization -- if already >95%, graphs won't help much.
- Look for gaps between kernel launches (CPU overhead) and many small kernels
(<50 us each). These are the targets for graphing.
- Annotate regions of interest to correlate idle GPU time with code:
with torch.cuda.nvtx.range("forward"):
output = model(input)
- Estimate benefit: count kernels per iteration. Workloads with hundreds of
small kernels and <80% GPU utilization are strong candidates.
Expected result: Identified bottleneck regions with low GPU occupancy between
kernels. Proceed to the appropriate workflow from the API Selection Guide.
Workflow 2: torch.compile(mode="reduce-overhead")
Goal: Automatic CUDA Graph capture with zero manual effort.
When to use: Quick experiment, unknown graph boundaries, already using
torch.compile.
Steps:
- Decorate the training step with
@torch.compile(mode="reduce-overhead"):@torch.compile(mode="reduce-overhead")
def train_step(model, x, target, criterion):
output = model(x)
loss = criterion(output, target)
loss.backward()
return loss
- Run the training loop normally -- graphs are captured automatically.
- Profile with Nsight Systems to see captured graphs:
nsys profile --cuda-graph-trace=graph python train.py
- If you see too many small graphs (graph fragmentation), check for graph
breaks:
.item(), print(), data-dependent control flow. Fix these or
escalate to Workflow 3+.
Trade-offs:
- Zero effort, but may create fragmented small graphs.
- Limited control over what gets graphed.
- Graph fragmentation limits performance gains compared to manual approaches.
Workflow 3: torch.cuda.make_graphed_callables()
Goal: Training with autograd support. Separate forward/backward graphs.
When to use: Training with custom loops, non-FP8, need autograd.
Steps:
- Prepare sample inputs matching training batch shape:
sample_input = torch.randn(batch_size, seq_len, hidden_size, device="cuda")
- Create the graphed model:
graphed_model = torch.cuda.make_graphed_callables(
model, (sample_input,), num_warmup_iters=3
)
- Use
graphed_model as a drop-in replacement in the training loop:for data, target in dataloader:
optimizer.zero_grad()
output = graphed_model(data)
loss = criterion(output, target)
loss.backward()
optimizer.step()
- If using AMP, set
cache_enabled=False:for data, target in dataloader:
optimizer.zero_grad()
with torch.amp.autocast("cuda", cache_enabled=False):
output = graphed_model(data)
loss = criterion(output, target)
loss.backward()
optimizer.step()
- If using DDP, construct DDP on a side stream and use 11 warmup iters:
os.environ["TORCH_NCCL_ASYNC_ERROR_HANDLING"] = "0"
s = torch.cuda.Stream()
with torch.cuda.stream(s):
model = DistributedDataParallel(model)
torch.cuda.current_stream().wait_stream(s)
graphed_model = torch.cuda.make_graphed_callables(
model, (sample_input,), num_warmup_iters=11
)
Limitations:
- No double backward (higher-order gradients).
- No module hooks during capture.
- Module structure is frozen after graphing (no add/remove parameters).
- Argument signature must match
sample_args exactly.
Workflow 4: TE make_graphed_callables
Goal: Per-callable graphing with FP8 support and pipeline parallelism.
When to use: FP8 training, PP with manual scheduling, non-Megatron models
needing FP8, or any PyTorch model that needs FP8-aware CUDA Graphs.
Steps:
- Import and configure:
from transformer_engine.pytorch.graph import make_graphed_callables
from transformer_engine.pytorch.fp8 import fp8_autocast
- Prepare sample inputs (one per callable per microbatch per chunk):
sample_args = tuple(
(torch.randn(batch_size, seq_len, hidden_size, device="cuda"),)
for _ in range(num_callables * num_microbatches)
)
- Define pipeline schedule if using PP (1-indexed chunk IDs, positive=fwd,
negative=bwd):
# Example: 2 chunks, 3 microbatches
layer_order = [1, 2, 1, 2, 1, 2, -2, -1, -2, -1, -2, -1]
- Wrap layers in CUDA Graphs:
graphed_layers = make_graphed_callables(
tuple(layers),
sample_args=sample_args,
fp8_enabled=True,
fp8_recipe=fp8_recipe,
fp8_weight_caching=True,
_order=layer_order, # None for no PP
)
- Training loop -- wrap with
fp8_autocast during replay:with fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):
for layer in graphed_layers[start:end]:
x = layer(x, is_first_microbatch=(mb_idx == 0))
# FP8 scaling auto-updated on fp8_autocast exit
optimizer.step()
Key points:
- AOT capture: Graphs captured before the training loop when you call
make_graphed_callables().
- Replay order must match
_order: The training loop must execute graphs
in the same interleaved order as specified during capture.
fp8_autocast required during replay: Without it, FP8 state is not
properly configured.
- Weight caching:
fp8_weight_caching=True caches FP8 weight
quantization across microbatches; pass is_first_microbatch kwarg to
control when weights are requantized.
For full API details, see references/api-te-megatron.md.
Workflow 5: MCore CudaGraphManager (Per-Layer)
Goal: Automatic per-layer graphing for Megatron-LM training.
When to use: Megatron-LM training, especially with PP > 1. Default choice
for Megatron users.
Steps:
- Enable via CLI flags (no code changes needed):
python pretrain_gpt.py \
--enable-cuda-graph \
--cuda-graph-num-warmup-steps 3
- Or enable via Python config:
config = TransformerConfig(
enable_cuda_graph=True,
cuda_graph_num_warmup_steps=3,
)
- Training loop is unchanged -- graphs are captured automatically after
warmup iterations.
Key points:
- Megatron layers only: Works with
TransformerLayer and MambaLayer.
- JIT capture: Records execution order during warmup, captures graphs
after warmup completes, then replays on subsequent iterations.
- Automatic FP8 handling: Uses
fp8_autocast(..., _graph=True) to skip
per-layer amax reduction; reduction happens once after all backward graphs.
- Automatic PP support: Handles microbatch interleaving automatically.
- Memory savings: Set
cuda_graph_share_io_buffers=True to share I/O
buffers between layers (requires no operations between layers).
- Memory pool strategy: Default uses separate pools per microbatch for
graph reuse. Set
cuda_graph_use_single_mempool=True for shared pool
(higher graph count but may reduce fragmentation).
Workflow 6: MCore FullCudaGraphWrapper (Full-Iteration)
Goal: Maximum performance. Captures forward+backward for all microbatches
as a single graph.
When to use: Maximum performance priority, static workloads, Megatron-LM
training.
Steps:
- Enable via CLI flags:
python pretrain_gpt.py \
--enable-cuda-graph \
--cuda-graph-scope full_iteration \
--cuda-graph-warmup-steps 1 \
--te-rng-tracker \
--no-check-for-nan-in-loss-and-grad
- Ensure all forward+backward code is capturable (no
.item(), no NaN
check, no dynamic control flow).
- Optimizer remains in eager mode by default (outside the graph). Can be
included inside the graph for maximum performance.
Key points:
- Only 2 graphs total: One for training, one for validation.
--te-rng-tracker required: Standard RNG uses CPU scalars that cannot
be captured; TE RNG uses device tensors compatible with graphs.
--no-check-for-nan-in-loss-and-grad mandatory: NaN checking uses
.item() which requires CPU-GPU sync, forbidden during capture.
- StaticBufferLoader: Pre-allocates input buffers for all microbatches
during warmup.
- Optimizer in/out of graph: Inside = maximum performance (all optimizer
kernels captured). Outside = more flexible (can change optimizer/LR without
recapture).
- JIT capture: Graph captured during training at iteration
warmup_steps + 1.
Workflow 7: torch.cuda.graph() (Manual)
Goal: Full control over capture and replay. Custom pipelines, full-iteration
capture without Megatron.
When to use: Need fine-grained control, non-Megatron full-iteration capture,
custom pipelines.
Inference pattern:
- Pre-allocate static input/output tensors:
static_input = torch.randn(batch_size, *shape, device="cuda")
- Warmup on a side stream (3 iterations, 11 for DDP):
s = torch.cuda.Stream()
with torch.cuda.stream(s):
for _ in range(3):
_ = model(static_input)
torch.cuda.current_stream().wait_stream(s)
- Capture the graph:
g = torch.cuda.CUDAGraph()
with torch.cuda.graph(g):
static_output = model(static_input)
- Replay loop -- update inputs via
.copy_(), clone outputs:for data in loader:
static_input.copy_(data)
g.replay()
result = static_output.clone()
Full training pattern (fwd+bwd+optimizer in one graph):
model = MyModel().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = torch.nn.CrossEntropyLoss()
static_input = torch.randn(batch_size, *shape, device="cuda")
static_target = torch.randint(0, num_classes, (batch_size,), device="cuda")
# Warmup
s = torch.cuda.Stream()
with torch.cuda.stream(s):
for _ in range(3):
optimizer.zero_grad()
with torch.amp.autocast("cuda", cache_enabled=False):
out = model(static_input)
loss = criterion(out, static_target)
loss.backward()
torch.cuda.current_stream().wait_stream(s)
# Capture
g = torch.cuda.CUDAGraph()
with torch.cuda.graph(g):
optimizer.zero_grad()
with torch.amp.autocast("cuda", cache_enabled=False):
static_output = model(static_input)
static_loss = criterion(static_output, static_target)
static_loss.backward()
# Replay loop
for data, target in loader:
static_input.copy_(data)
static_target.copy_(target)
g.replay()
optimizer.step()
DDP setup:
os.environ["TORCH_NCCL_ASYNC_ERROR_HANDLING"] = "0"
s = torch.cuda.Stream()
with torch.cuda.stream(s):
model = DistributedDataParallel(model)
# 11 warmup iterations for DDP
with torch.cuda.stream(s):
for _ in range(11):
out = model(static_input)
out.sum().backward()
torch.cuda.current_stream().wait_stream(s)
# Capture on the same side stream
with torch.cuda.graph(g):
static_output = model(static_input)
Memory pool sharing for multiple graphs:
g1 = torch.cuda.CUDAGraph()
with torch.cuda.graph(g1):
out1 = model_a(static_in_a)
# Second graph shares first graph's memory pool
g2 = torch.cuda.CUDAGraph()
with torch.cuda.graph(g2, pool=g1.pool()):
out2 = model_b(static_in_b)
Custom RNG registration:
gen = torch.cuda.default_generators[0]
g = torch.cuda.CUDAGraph()
g.register_generator_state(gen)
with torch.cuda.graph(g):
out = model(static_input) # RNG state properly captured
Navigating Between Workflows
- torch.compile gives insufficient speedup --> escalate to
make_graphed_callables (Workflow 3) for larger, fewer graphs.
- make_graphed_callables can't handle FP8/PP --> TE
make_graphed_callables (Workflow 4).
- Need Megatron per-layer automatic --> CudaGraphManager (Workflow 5).
- Want maximum perf --> FullCudaGraphWrapper (Workflow 6) or manual
full-iteration capture (Workflow 7).
- Something too hard to graph --> partial capture (graph what you can,
leave the rest in eager mode).
- User wants best absolute perf --> skip directly to Workflow 6
(Megatron) or Workflow 7 (manual).
- Start small, expand progressively: Begin with one module/layer. Verify
correctness. Then expand to more layers, full forward pass, add backward,
and eventually full iteration with optimizer.
Making Code Graph-Compatible
These principles apply to all workflows. Code inside the captured region must
satisfy three constraints.
Principle 1: GPU-Only
Only GPU operations are captured. CPU-side code (Python logic, I/O, logging)
executes during capture but is eliminated during replay.
Violations:
- File I/O:
data = torch.load("file.pt") won't reload on replay
- CPU preprocessing:
tokens = tokenizer.encode(text) won't re-tokenize
- Logging:
print(f"Step {i}") won't print during replay
- CPU RNG:
random.randint(0, 10) won't regenerate
- CPU bookkeeping:
buffer.append(tensor) won't populate during replay
Fix: Move all CPU-side operations outside the graphed region.
Principle 2: Sync-Free
No CPU-GPU synchronization inside the graph. The CPU queues work continuously
without waiting for GPU results.
Violations:
.item() to get scalar values
.cpu() to move tensors for inspection
torch.cuda.synchronize() or stream.synchronize()
print(tensor) (implicitly syncs)
Fix: Invoke the perf-torch-sync-free skill for systematic detection and
elimination of sync points. Use torch.cuda.set_sync_debug_mode("warn") to
find hidden syncs.
Principle 3: Static
All operations, control flow, memory addresses, and shapes must be fixed
across all replays.
Violations and fixes:
| Dynamic aspect |
Fix |
if loss > threshold: |
torch.where(condition, a, b) |
input = new_tensor (address changes) |
Pre-allocate + .copy_() |
| Python scalars (lr, temperature) |
GPU tensor + .fill_() |
| Variable batch size / sequence length |
Padding or bucketing |
| MoE / dynamic routing |
Partial graphing |
For detailed patterns, see references/patterns-dynamic.md.
Compatibility Checklist
Verify every item before attempting capture:
For the complete checklist with references, see references/patterns-compatibility.md.
Output Formats
Success indicators:
g.replay() completes without errors
- Outputs match eager mode within tolerance (
torch.allclose)
- Nsight Systems profile shows single graph launch replacing many kernels
- GPU utilization increases, training/inference latency decreases
Key metrics:
| Metric |
How to Check |
| Correctness |
torch.allclose(eager, graphed, rtol=1e-5) |
| Speedup |
Wall-clock time comparison |
| GPU utilization |
nvidia-smi or Nsight Systems timeline |
| Memory overhead |
torch.cuda.memory_summary() |
Error Handling
| Error |
Cause |
Fix |
StreamCaptureUnsupported (900) |
Sync op during capture (.item(), .cpu()) |
Move sync outside graph |
StreamCaptureInvalidated (901) |
Background thread (e.g., pin_memory) |
capture_error_mode="thread_local" |
StreamCaptureUnjoined (904) |
Side stream didn't rejoin capture stream |
capture_stream.wait_stream(side_stream) |
StreamCaptureImplicit (906) |
AccumulateGrad on default stream |
Warmup on side stream before capture |
| Illegal memory access |
Input tensor freed/reassigned |
Keep persistent ref, use .copy_() |
| Wrong numerical results |
Dynamic behavior frozen at capture |
See references/patterns-compatibility.md |
| OOM with multiple graphs |
Pools can't share memory |
pool=g1.pool() for sequential graphs |
| No speedup |
Already GPU-bound or wrong capture scope |
Profile with nsys first (Workflow 1) |
| FP8 scaling corruption |
TE without fp8_autocast during replay |
Wrap with fp8_autocast(enabled=True) |
| PP replay order mismatch |
Wrong execution order during replay |
Match _order / capture sequence exactly |
| FullCudaGraphWrapper capture fail |
NaN check or sync enabled |
--no-check-for-nan-in-loss-and-grad |
| RNG failure with FullCudaGraphWrapper |
Standard RNG not capturable |
--te-rng-tracker |
| DDP capture failure |
Async error handling watchdog |
TORCH_NCCL_ASYNC_ERROR_HANDLING=0 |
| DDP AccumulateGrad on default stream |
DDP constructed on default stream |
Construct DDP in side stream context |
| Autocast cache invalidation |
Cached cast tensors freed on exit |
cache_enabled=False |
For detailed troubleshooting, see references/troubleshooting.md.
Finding More Information
Use this 3-tier lookup hierarchy -- start at Tier 1 and escalate only when
needed.
Tier 1: This File (SKILL.md)
You are reading it now. The workflows, compatibility checklist, and error
table above cover the most common tasks. Search this file first before going
deeper.
Tier 2: references/ Directory
The references/ directory beside this file contains distilled reference
material -- API details, patterns, and troubleshooting pages.
How to search:
- Grep for your keyword across
references/ -- headers are designed to be
grep-friendly.
- Read only the file that grep points you to. Do not read every file.
Available references:
references/api-pytorch.md -- PyTorch CUDA Graph APIs (torch.cuda.graph,
make_graphed_callables, torch.compile reduce-overhead)
references/api-te-megatron.md -- TE make_graphed_callables,
CudaGraphManager, FullCudaGraphWrapper implementations
references/patterns-compatibility.md -- GPU-only, sync-free, and static
principles with full checklist
references/patterns-dynamic.md -- Dynamic control flow, tensors, scalars,
shapes: workarounds and patterns
references/troubleshooting.md -- Capture failures, numerical errors,
memory issues, performance issues
Tier 3: Original Documentation
If Tiers 1-2 do not answer the question, consult the original sources:
- NVIDIA guide:
https://docs.nvidia.com/dl-cuda-graph/latest/index.html
- PyTorch docs:
https://docs.pytorch.org/docs/stable/notes/cuda.html
(CUDA Graphs section)
- TE docs:
https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/index.html
- Megatron Core docs:
https://docs.nvidia.com/megatron-core/developer-guide/latest/index.html
Return to Tier 2 afterward and consider whether the answer should be distilled
into the references directory for next time.
1---2name: perf-torch-cuda-graphs3description: >-4 Apply CUDA Graphs to PyTorch workloads — API selection (torch.compile, PyTorch5 make_graphed_callables, TE make_graphed_callables, MCore CudaGraphManager,6 FullCudaGraphWrapper, manual torch.cuda.graph), code compatibility, capture7 workflows, dynamic pattern handling, and troubleshooting.8 Triggers: CUDA graph, torch.cuda.graph, make_graphed_callables, reduce-overhead,9 graph capture, graph replay, kernel launch overhead, CudaGraphManager,10 FullCudaGraphWrapper, full-iteration graph, stream capture.11tags:12 - cuda-graph13 - optimization14 - pytorch15license: Apache-2.016metadata:17 source: https://skillmd.com/skills/wenyi-li/perf-torch-cuda-graphs18 author: NVIDIA Corporation19---2021# CUDA Graphs for PyTorch2223CUDA Graphs capture a sequence of GPU operations once and replay them with24minimal CPU overhead. This skill guides applying CUDA Graphs to PyTorch25training and inference workloads using native PyTorch APIs, Transformer26Engine, and Megatron-LM.2728## When to Use2930Reach for this skill when you encounter:3132- **Triggers**: User wants to optimize with CUDA Graphs, reduce kernel launch33 overhead, or speed up training/inference loops34- **Symptoms**: Low GPU utilization (<80%), many small kernel launches (<50 us35 each), CPU-bound training, high kernel launch latency visible in Nsight36 Systems profiles37- **Keywords**: "CUDA graph", "torch.cuda.graph", "make_graphed_callables",38 "reduce-overhead", "graph capture", "graph replay", "kernel launch overhead",39 "CudaGraphManager", "FullCudaGraphWrapper", "full-iteration graph", "stream40 capture"4142Do NOT use this skill for:4344- General PyTorch performance tuning unrelated to kernel launch overhead45- CUDA kernel development or custom CUDA C++ code46- Host-device sync elimination only (use **perf-torch-sync-free** skill instead)47- Nsight Systems profiling (use **perf-nsight-systems** skill)48- TensorFlow/JAX graph compilation (different APIs entirely)4950## Requirements5152| Dependency | Version | Notes |53|------------|---------|-------|54| PyTorch | >= 1.10 | `torch.cuda.graph()` available |55| CUDA | >= 11.0 | Graph update APIs |56| GPU | NVIDIA (any) | Required for CUDA |57| Nsight Systems | any | Optional, for profiling |58| APEX | any | Optional, for capturable optimizers |59| Transformer Engine | >= 2.2 | Optional, for FP8-aware graphing |60| Megatron-LM | core >= 0.14.0 | Optional, for CudaGraphManager / FullCudaGraphWrapper |6162## API Selection Guide6364Choose the API based on your framework and performance needs.6566| Situation | API | Workflow |67|-----------|-----|---------|68| Quick experiment, unknown graph boundaries | `torch.compile(mode="reduce-overhead")` | Workflow 2 |69| Training, need autograd, no FP8/PP | `torch.cuda.make_graphed_callables()` | Workflow 3 |70| Any PyTorch model, FP8 or PP support | TE `make_graphed_callables` | Workflow 4 |71| Megatron-LM, per-layer, automatic | MCore `CudaGraphManager` | Workflow 5 |72| Maximum perf, full-iteration capture | MCore `FullCudaGraphWrapper` | Workflow 6 |73| Full manual control, custom pipelines | `torch.cuda.graph()` | Workflow 7 |7475**Decision flowchart:**76771. Using Megatron-LM with FP8/PP?78 - Yes, want maximum perf with static workload --> Workflow 6 (FullCudaGraphWrapper)79 - Yes, want per-layer automatic graphing --> Workflow 5 (CudaGraphManager)80 - Yes, want manual control over what gets graphed --> Workflow 4 (TE make_graphed_callables)812. Using Transformer Engine without Megatron?82 - Yes, need FP8 or PP --> Workflow 4 (TE make_graphed_callables)833. General PyTorch?84 - Want zero effort, okay with fragmented graphs --> Workflow 2 (torch.compile)85 - Want autograd support, training loop --> Workflow 3 (PyTorch make_graphed_callables)86 - Want full manual control --> Workflow 7 (torch.cuda.graph)8788**Strategy:** Start with the highest-level API available for your framework.89Move to lower-level APIs only if you need more control, hit limitations, or90do not achieve the expected performance improvement.9192## Workflows9394### Workflow 1: Profile and Decide Whether Graphs Help9596Goal: Determine if CUDA Graphs will benefit your workload before investing97effort.98991. Profile with Nsight Systems:100 ```bash101 nsys profile --cuda-graph-trace=graph python train.py102 ```1032. Check GPU utilization -- if already >95%, graphs won't help much.1043. Look for gaps between kernel launches (CPU overhead) and many small kernels105 (<50 us each). These are the targets for graphing.1064. Annotate regions of interest to correlate idle GPU time with code:107 ```python108 with torch.cuda.nvtx.range("forward"):109 output = model(input)110 ```1115. Estimate benefit: count kernels per iteration. Workloads with hundreds of112 small kernels and <80% GPU utilization are strong candidates.113114Expected result: Identified bottleneck regions with low GPU occupancy between115kernels. Proceed to the appropriate workflow from the API Selection Guide.116117### Workflow 2: torch.compile(mode="reduce-overhead")118119Goal: Automatic CUDA Graph capture with zero manual effort.120121When to use: Quick experiment, unknown graph boundaries, already using122`torch.compile`.123124Steps:1251261. Decorate the training step with `@torch.compile(mode="reduce-overhead")`:127 ```python128 @torch.compile(mode="reduce-overhead")129 def train_step(model, x, target, criterion):130 output = model(x)131 loss = criterion(output, target)132 loss.backward()133 return loss134 ```1352. Run the training loop normally -- graphs are captured automatically.1363. Profile with Nsight Systems to see captured graphs:137 ```bash138 nsys profile --cuda-graph-trace=graph python train.py139 ```1404. If you see too many small graphs (graph fragmentation), check for graph141 breaks: `.item()`, `print()`, data-dependent control flow. Fix these or142 escalate to Workflow 3+.143144Trade-offs:145- Zero effort, but may create fragmented small graphs.146- Limited control over what gets graphed.147- Graph fragmentation limits performance gains compared to manual approaches.148149### Workflow 3: torch.cuda.make_graphed_callables()150151Goal: Training with autograd support. Separate forward/backward graphs.152153When to use: Training with custom loops, non-FP8, need autograd.154155Steps:1561571. Prepare sample inputs matching training batch shape:158 ```python159 sample_input = torch.randn(batch_size, seq_len, hidden_size, device="cuda")160 ```1612. Create the graphed model:162 ```python163 graphed_model = torch.cuda.make_graphed_callables(164 model, (sample_input,), num_warmup_iters=3165 )166 ```1673. Use `graphed_model` as a drop-in replacement in the training loop:168 ```python169 for data, target in dataloader:170 optimizer.zero_grad()171 output = graphed_model(data)172 loss = criterion(output, target)173 loss.backward()174 optimizer.step()175 ```1764. If using AMP, set `cache_enabled=False`:177 ```python178 for data, target in dataloader:179 optimizer.zero_grad()180 with torch.amp.autocast("cuda", cache_enabled=False):181 output = graphed_model(data)182 loss = criterion(output, target)183 loss.backward()184 optimizer.step()185 ```1865. If using DDP, construct DDP on a side stream and use 11 warmup iters:187 ```python188 os.environ["TORCH_NCCL_ASYNC_ERROR_HANDLING"] = "0"189 s = torch.cuda.Stream()190 with torch.cuda.stream(s):191 model = DistributedDataParallel(model)192 torch.cuda.current_stream().wait_stream(s)193194 graphed_model = torch.cuda.make_graphed_callables(195 model, (sample_input,), num_warmup_iters=11196 )197 ```198199Limitations:200- No double backward (higher-order gradients).201- No module hooks during capture.202- Module structure is frozen after graphing (no add/remove parameters).203- Argument signature must match `sample_args` exactly.204205### Workflow 4: TE make_graphed_callables206207Goal: Per-callable graphing with FP8 support and pipeline parallelism.208209When to use: FP8 training, PP with manual scheduling, non-Megatron models210needing FP8, or any PyTorch model that needs FP8-aware CUDA Graphs.211212Steps:2132141. Import and configure:215 ```python216 from transformer_engine.pytorch.graph import make_graphed_callables217 from transformer_engine.pytorch.fp8 import fp8_autocast218 ```2192. Prepare sample inputs (one per callable per microbatch per chunk):220 ```python221 sample_args = tuple(222 (torch.randn(batch_size, seq_len, hidden_size, device="cuda"),)223 for _ in range(num_callables * num_microbatches)224 )225 ```2263. Define pipeline schedule if using PP (1-indexed chunk IDs, positive=fwd,227 negative=bwd):228 ```python229 # Example: 2 chunks, 3 microbatches230 layer_order = [1, 2, 1, 2, 1, 2, -2, -1, -2, -1, -2, -1]231 ```2324. Wrap layers in CUDA Graphs:233 ```python234 graphed_layers = make_graphed_callables(235 tuple(layers),236 sample_args=sample_args,237 fp8_enabled=True,238 fp8_recipe=fp8_recipe,239 fp8_weight_caching=True,240 _order=layer_order, # None for no PP241 )242 ```2435. Training loop -- wrap with `fp8_autocast` during replay:244 ```python245 with fp8_autocast(enabled=True, fp8_recipe=fp8_recipe):246 for layer in graphed_layers[start:end]:247 x = layer(x, is_first_microbatch=(mb_idx == 0))248 # FP8 scaling auto-updated on fp8_autocast exit249 optimizer.step()250 ```251252Key points:253- **AOT capture**: Graphs captured before the training loop when you call254 `make_graphed_callables()`.255- **Replay order must match `_order`**: The training loop must execute graphs256 in the same interleaved order as specified during capture.257- **`fp8_autocast` required during replay**: Without it, FP8 state is not258 properly configured.259- **Weight caching**: `fp8_weight_caching=True` caches FP8 weight260 quantization across microbatches; pass `is_first_microbatch` kwarg to261 control when weights are requantized.262263For full API details, see `references/api-te-megatron.md`.264265### Workflow 5: MCore CudaGraphManager (Per-Layer)266267Goal: Automatic per-layer graphing for Megatron-LM training.268269When to use: Megatron-LM training, especially with PP > 1. Default choice270for Megatron users.271272Steps:2732741. Enable via CLI flags (no code changes needed):275 ```bash276 python pretrain_gpt.py \277 --enable-cuda-graph \278 --cuda-graph-num-warmup-steps 3279 ```2802. Or enable via Python config:281 ```python282 config = TransformerConfig(283 enable_cuda_graph=True,284 cuda_graph_num_warmup_steps=3,285 )286 ```2873. Training loop is unchanged -- graphs are captured automatically after288 warmup iterations.289290Key points:291- **Megatron layers only**: Works with `TransformerLayer` and `MambaLayer`.292- **JIT capture**: Records execution order during warmup, captures graphs293 after warmup completes, then replays on subsequent iterations.294- **Automatic FP8 handling**: Uses `fp8_autocast(..., _graph=True)` to skip295 per-layer amax reduction; reduction happens once after all backward graphs.296- **Automatic PP support**: Handles microbatch interleaving automatically.297- **Memory savings**: Set `cuda_graph_share_io_buffers=True` to share I/O298 buffers between layers (requires no operations between layers).299- **Memory pool strategy**: Default uses separate pools per microbatch for300 graph reuse. Set `cuda_graph_use_single_mempool=True` for shared pool301 (higher graph count but may reduce fragmentation).302303### Workflow 6: MCore FullCudaGraphWrapper (Full-Iteration)304305Goal: Maximum performance. Captures forward+backward for all microbatches306as a single graph.307308When to use: Maximum performance priority, static workloads, Megatron-LM309training.310311Steps:3123131. Enable via CLI flags:314 ```bash315 python pretrain_gpt.py \316 --enable-cuda-graph \317 --cuda-graph-scope full_iteration \318 --cuda-graph-warmup-steps 1 \319 --te-rng-tracker \320 --no-check-for-nan-in-loss-and-grad321 ```3222. Ensure all forward+backward code is capturable (no `.item()`, no NaN323 check, no dynamic control flow).3243. Optimizer remains in eager mode by default (outside the graph). Can be325 included inside the graph for maximum performance.326327Key points:328- **Only 2 graphs total**: One for training, one for validation.329- **`--te-rng-tracker` required**: Standard RNG uses CPU scalars that cannot330 be captured; TE RNG uses device tensors compatible with graphs.331- **`--no-check-for-nan-in-loss-and-grad` mandatory**: NaN checking uses332 `.item()` which requires CPU-GPU sync, forbidden during capture.333- **StaticBufferLoader**: Pre-allocates input buffers for all microbatches334 during warmup.335- **Optimizer in/out of graph**: Inside = maximum performance (all optimizer336 kernels captured). Outside = more flexible (can change optimizer/LR without337 recapture).338- **JIT capture**: Graph captured during training at iteration339 `warmup_steps + 1`.340341### Workflow 7: torch.cuda.graph() (Manual)342343Goal: Full control over capture and replay. Custom pipelines, full-iteration344capture without Megatron.345346When to use: Need fine-grained control, non-Megatron full-iteration capture,347custom pipelines.348349**Inference pattern:**3503511. Pre-allocate static input/output tensors:352 ```python353 static_input = torch.randn(batch_size, *shape, device="cuda")354 ```3552. Warmup on a side stream (3 iterations, 11 for DDP):356 ```python357 s = torch.cuda.Stream()358 with torch.cuda.stream(s):359 for _ in range(3):360 _ = model(static_input)361 torch.cuda.current_stream().wait_stream(s)362 ```3633. Capture the graph:364 ```python365 g = torch.cuda.CUDAGraph()366 with torch.cuda.graph(g):367 static_output = model(static_input)368 ```3694. Replay loop -- update inputs via `.copy_()`, clone outputs:370 ```python371 for data in loader:372 static_input.copy_(data)373 g.replay()374 result = static_output.clone()375 ```376377**Full training pattern (fwd+bwd+optimizer in one graph):**378379```python380model = MyModel().cuda()381optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)382criterion = torch.nn.CrossEntropyLoss()383384static_input = torch.randn(batch_size, *shape, device="cuda")385static_target = torch.randint(0, num_classes, (batch_size,), device="cuda")386387# Warmup388s = torch.cuda.Stream()389with torch.cuda.stream(s):390 for _ in range(3):391 optimizer.zero_grad()392 with torch.amp.autocast("cuda", cache_enabled=False):393 out = model(static_input)394 loss = criterion(out, static_target)395 loss.backward()396torch.cuda.current_stream().wait_stream(s)397398# Capture399g = torch.cuda.CUDAGraph()400with torch.cuda.graph(g):401 optimizer.zero_grad()402 with torch.amp.autocast("cuda", cache_enabled=False):403 static_output = model(static_input)404 static_loss = criterion(static_output, static_target)405 static_loss.backward()406407# Replay loop408for data, target in loader:409 static_input.copy_(data)410 static_target.copy_(target)411 g.replay()412 optimizer.step()413```414415**DDP setup:**416417```python418os.environ["TORCH_NCCL_ASYNC_ERROR_HANDLING"] = "0"419420s = torch.cuda.Stream()421with torch.cuda.stream(s):422 model = DistributedDataParallel(model)423424# 11 warmup iterations for DDP425with torch.cuda.stream(s):426 for _ in range(11):427 out = model(static_input)428 out.sum().backward()429torch.cuda.current_stream().wait_stream(s)430431# Capture on the same side stream432with torch.cuda.graph(g):433 static_output = model(static_input)434```435436**Memory pool sharing for multiple graphs:**437438```python439g1 = torch.cuda.CUDAGraph()440with torch.cuda.graph(g1):441 out1 = model_a(static_in_a)442443# Second graph shares first graph's memory pool444g2 = torch.cuda.CUDAGraph()445with torch.cuda.graph(g2, pool=g1.pool()):446 out2 = model_b(static_in_b)447```448449**Custom RNG registration:**450451```python452gen = torch.cuda.default_generators[0]453g = torch.cuda.CUDAGraph()454g.register_generator_state(gen)455with torch.cuda.graph(g):456 out = model(static_input) # RNG state properly captured457```458459### Navigating Between Workflows460461- **torch.compile gives insufficient speedup** --> escalate to462 `make_graphed_callables` (Workflow 3) for larger, fewer graphs.463- **make_graphed_callables can't handle FP8/PP** --> TE464 `make_graphed_callables` (Workflow 4).465- **Need Megatron per-layer automatic** --> CudaGraphManager (Workflow 5).466- **Want maximum perf** --> FullCudaGraphWrapper (Workflow 6) or manual467 full-iteration capture (Workflow 7).468- **Something too hard to graph** --> partial capture (graph what you can,469 leave the rest in eager mode).470- **User wants best absolute perf** --> skip directly to Workflow 6471 (Megatron) or Workflow 7 (manual).472- **Start small, expand progressively**: Begin with one module/layer. Verify473 correctness. Then expand to more layers, full forward pass, add backward,474 and eventually full iteration with optimizer.475476## Making Code Graph-Compatible477478These principles apply to all workflows. Code inside the captured region must479satisfy three constraints.480481### Principle 1: GPU-Only482483Only GPU operations are captured. CPU-side code (Python logic, I/O, logging)484executes during capture but is eliminated during replay.485486Violations:487- File I/O: `data = torch.load("file.pt")` won't reload on replay488- CPU preprocessing: `tokens = tokenizer.encode(text)` won't re-tokenize489- Logging: `print(f"Step {i}")` won't print during replay490- CPU RNG: `random.randint(0, 10)` won't regenerate491- CPU bookkeeping: `buffer.append(tensor)` won't populate during replay492493Fix: Move all CPU-side operations outside the graphed region.494495### Principle 2: Sync-Free496497No CPU-GPU synchronization inside the graph. The CPU queues work continuously498without waiting for GPU results.499500Violations:501- `.item()` to get scalar values502- `.cpu()` to move tensors for inspection503- `torch.cuda.synchronize()` or `stream.synchronize()`504- `print(tensor)` (implicitly syncs)505506Fix: **Invoke the perf-torch-sync-free skill** for systematic detection and507elimination of sync points. Use `torch.cuda.set_sync_debug_mode("warn")` to508find hidden syncs.509510### Principle 3: Static511512All operations, control flow, memory addresses, and shapes must be fixed513across all replays.514515Violations and fixes:516517| Dynamic aspect | Fix |518|---------------|-----|519| `if loss > threshold:` | `torch.where(condition, a, b)` |520| `input = new_tensor` (address changes) | Pre-allocate + `.copy_()` |521| Python scalars (lr, temperature) | GPU tensor + `.fill_()` |522| Variable batch size / sequence length | Padding or bucketing |523| MoE / dynamic routing | Partial graphing |524525For detailed patterns, see `references/patterns-dynamic.md`.526527### Compatibility Checklist528529Verify every item before attempting capture:530531- [ ] No `.item()`, `.cpu()`, `.numpy()`, `print(tensor)` inside graph532- [ ] No `torch.cuda.synchronize()` or `stream.synchronize()`533- [ ] No `if tensor_value:` -- use `torch.where()` instead534- [ ] All inputs pre-allocated, updated via `.copy_()`535- [ ] All shapes fixed (use padding or bucketing for variable sizes)536- [ ] Python scalars --> GPU tensors with `.fill_()`537- [ ] Output tensors `.clone()`d before next replay538- [ ] `cache_enabled=False` with `torch.amp.autocast`539- [ ] Custom RNG generators registered with `graph.register_generator_state()`540- [ ] Use `graphsafe_get_state()` / `graphsafe_set_state()` for RNG541- [ ] Warmup completed (3 standard, 11 for DDP)542- [ ] DDP: `TORCH_NCCL_ASYNC_ERROR_HANDLING=0`, construct on side stream543- [ ] DDP: NCCL >= 2.9.6 for full graph capture544- [ ] Libraries/extensions use `torch.cuda.current_stream()`, not default stream545- [ ] No pinned memory allocation during capture (triggers hidden event query)546- [ ] `activation_checkpointing`: `preserve_rng_state=False`547- [ ] Global tensors used in graph kept alive (not deleted/reassigned)548- [ ] No `torch.compile` functions inside manual capture without prior warmup549- [ ] Gradient clipping uses sync-free `clip_grad_norm_` (PyTorch >= 1.13)550551For the complete checklist with references, see `references/patterns-compatibility.md`.552553## Output Formats554555**Success indicators:**556- `g.replay()` completes without errors557- Outputs match eager mode within tolerance (`torch.allclose`)558- Nsight Systems profile shows single graph launch replacing many kernels559- GPU utilization increases, training/inference latency decreases560561**Key metrics:**562563| Metric | How to Check |564|--------|-------------|565| Correctness | `torch.allclose(eager, graphed, rtol=1e-5)` |566| Speedup | Wall-clock time comparison |567| GPU utilization | `nvidia-smi` or Nsight Systems timeline |568| Memory overhead | `torch.cuda.memory_summary()` |569570## Error Handling571572| Error | Cause | Fix |573|-------|-------|-----|574| `StreamCaptureUnsupported` (900) | Sync op during capture (`.item()`, `.cpu()`) | Move sync outside graph |575| `StreamCaptureInvalidated` (901) | Background thread (e.g., pin_memory) | `capture_error_mode="thread_local"` |576| `StreamCaptureUnjoined` (904) | Side stream didn't rejoin capture stream | `capture_stream.wait_stream(side_stream)` |577| `StreamCaptureImplicit` (906) | AccumulateGrad on default stream | Warmup on side stream before capture |578| Illegal memory access | Input tensor freed/reassigned | Keep persistent ref, use `.copy_()` |579| Wrong numerical results | Dynamic behavior frozen at capture | See `references/patterns-compatibility.md` |580| OOM with multiple graphs | Pools can't share memory | `pool=g1.pool()` for sequential graphs |581| No speedup | Already GPU-bound or wrong capture scope | Profile with nsys first (Workflow 1) |582| FP8 scaling corruption | TE without `fp8_autocast` during replay | Wrap with `fp8_autocast(enabled=True)` |583| PP replay order mismatch | Wrong execution order during replay | Match `_order` / capture sequence exactly |584| FullCudaGraphWrapper capture fail | NaN check or sync enabled | `--no-check-for-nan-in-loss-and-grad` |585| RNG failure with FullCudaGraphWrapper | Standard RNG not capturable | `--te-rng-tracker` |586| DDP capture failure | Async error handling watchdog | `TORCH_NCCL_ASYNC_ERROR_HANDLING=0` |587| DDP AccumulateGrad on default stream | DDP constructed on default stream | Construct DDP in side stream context |588| Autocast cache invalidation | Cached cast tensors freed on exit | `cache_enabled=False` |589590For detailed troubleshooting, see `references/troubleshooting.md`.591592## Finding More Information593594Use this 3-tier lookup hierarchy -- start at Tier 1 and escalate only when595needed.596597### Tier 1: This File (SKILL.md)598599You are reading it now. The workflows, compatibility checklist, and error600table above cover the most common tasks. Search this file first before going601deeper.602603### Tier 2: references/ Directory604605The `references/` directory beside this file contains distilled reference606material -- API details, patterns, and troubleshooting pages.607608**How to search:**6091. Grep for your keyword across `references/` -- headers are designed to be610 grep-friendly.6112. Read only the file that grep points you to. Do not read every file.612613Available references:614- `references/api-pytorch.md` -- PyTorch CUDA Graph APIs (`torch.cuda.graph`,615 `make_graphed_callables`, `torch.compile reduce-overhead`)616- `references/api-te-megatron.md` -- TE `make_graphed_callables`,617 CudaGraphManager, FullCudaGraphWrapper implementations618- `references/patterns-compatibility.md` -- GPU-only, sync-free, and static619 principles with full checklist620- `references/patterns-dynamic.md` -- Dynamic control flow, tensors, scalars,621 shapes: workarounds and patterns622- `references/troubleshooting.md` -- Capture failures, numerical errors,623 memory issues, performance issues624625### Tier 3: Original Documentation626627If Tiers 1-2 do not answer the question, consult the original sources:628- **NVIDIA guide**: `https://docs.nvidia.com/dl-cuda-graph/latest/index.html`629- **PyTorch docs**: `https://docs.pytorch.org/docs/stable/notes/cuda.html`630 (CUDA Graphs section)631- **TE docs**: `https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/index.html`632- **Megatron Core docs**: `https://docs.nvidia.com/megatron-core/developer-guide/latest/index.html`633634Return to Tier 2 afterward and consider whether the answer should be distilled635into the references directory for next time.