Workload Profiling
Quick Reference
Pick ONE path based on the workload type:
| Workload |
Approach |
Section |
| Training loop |
Manual torch.cuda.synchronize() + time.perf_counter() with warmup |
Loop Workloads — Manual Timing |
| Single kernel or op |
Write CUDA event benchmark (pre-allocate, warmup, event pairs) |
Non-Loop Workloads — CUDA Event Benchmarking |
| Add timeline labels for nsys |
Use @nvtx.annotate decorator or context manager |
NVTX Reference |
Principles
- Measure, don't guess. Every performance claim must trace back to profiler output or structured measurement data. Never invent metrics.
- Isolate steady-state. Warmup costs (CUDA context init, cuDNN autotuning, JIT compilation) distort measurements. Always exclude warmup iterations before collecting data.
- Use hardware timing. CUDA events measure GPU time precisely. CPU timers (
time.perf_counter()) include host overhead and miss asynchronous execution.
- No sync inside measurement loops. Each
torch.cuda.synchronize() adds 10-50us overhead. Record CUDA events asynchronously, sync once at the end.
- Pre-allocate everything. Tensors, events, compiled kernels — all before the timing loop. For CuTe DSL kernels, pre-compile with
cute.compile().
- Minimize profiler interference. Start with lightweight measurement (manual timing for latency/throughput) and escalate to heavier tools (Kineto, nsys, ncu) only when lighter tools cannot answer the question.
Loop Workloads — Manual Timing
For training loops and iterative workloads, use manual torch.cuda.synchronize() + time.perf_counter() timing with warmup to measure per-iteration latency, throughput, and data load time.
Injection Template
Read the user's training script, understand the dataloader and loop structure, then inject timing code.
import time
import torch
WARMUP = 5
NUM_ITERS = 30
BATCH_SIZE = 128 # global batch size for throughput calculation
iter_times = []
data_times = []
for i, batch in enumerate(dataloader):
if i >= WARMUP + NUM_ITERS:
break
t_data_end = time.perf_counter()
torch.cuda.synchronize()
t_start = time.perf_counter()
# ... existing training loop body ...
torch.cuda.synchronize()
t_end = time.perf_counter()
if i >= WARMUP:
iter_ms = (t_end - t_start) * 1000
iter_times.append(iter_ms)
if i > 0:
data_times.append((t_data_end - prev_iter_end) * 1000)
print(f"[{i:04d}]: iter {iter_ms:.2f} ms, fps {BATCH_SIZE / (iter_ms / 1000):.2f}")
prev_iter_end = t_end
import statistics
print(f"Average: iter {statistics.mean(iter_times):.2f} ms, "
f"fps {BATCH_SIZE / (statistics.mean(iter_times) / 1000):.2f}")
Interpreting Results
- iter (ms): Wall-clock time per iteration (compute + communication, excluding data loading)
- data (ms): Time spent in dataloader between iterations. If
data / iter > 0.2, data loading is a bottleneck.
- fps: Global throughput in samples/second. Use with known FLOPs-per-sample to compute MFU.
Limitations
Manual timing reports aggregate iteration timing — not per-sub-phase breakdown (forward, backward, optimizer). When the user asks where time is spent within compute:
- Add
torch.cuda.synchronize() + time.perf_counter() around each sub-phase for a one-off diagnosis, OR
- Add NVTX annotations and run with
nsys profile for timeline visualization.
Non-Loop Workloads — CUDA Event Benchmarking
For single kernels, one-shot inference, or standalone operations, write CUDA event benchmarking code directly.
PyTorch: Simple (Mean Only)
import torch
def benchmark(fn, warmup=50, iters=100):
for _ in range(warmup):
fn()
torch.cuda.synchronize()
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for _ in range(iters):
fn()
end.record()
torch.cuda.synchronize()
return start.elapsed_time(end) / iters # ms per iteration
PyTorch: Detailed (Per-Iteration Stats)
import torch
import statistics
def benchmark_detailed(fn, warmup=50, iters=100):
for _ in range(warmup):
fn()
torch.cuda.synchronize()
starts = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
ends = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
for i in range(iters):
starts[i].record()
fn()
ends[i].record()
torch.cuda.synchronize()
times = [starts[i].elapsed_time(ends[i]) for i in range(iters)]
return {
"mean_ms": statistics.mean(times),
"median_ms": statistics.median(times),
"std_ms": statistics.stdev(times) if len(times) > 1 else 0,
"min_ms": min(times),
"max_ms": max(times),
}
Anti-Patterns
| Anti-Pattern |
Problem |
torch.cuda.synchronize() before AND after each iteration |
Adds ~10-50us overhead per iteration |
time.perf_counter() for GPU timing |
Measures CPU time, misses async GPU execution |
| Missing warmup |
First iterations include JIT, clock ramp-up, context init |
| Allocating tensors inside measurement loop |
Allocation overhead pollutes timing |
| Reporting only mean |
Hides variance, outliers, bimodal distributions |
For additional benchmarking templates (CUDA Graph, CuTe DSL, Triton, Raw CUDA), see references/benchmarking-patterns.md.
NVTX Reference
NVTX (NVIDIA Tools Extension) adds named annotations to profiler timelines. Use NVTX to label phases (forward, backward, optimizer) for readability in nsys — not for measurement.
import nvtx
# Decorator — annotates every call
@nvtx.annotate("training_step", color="blue")
def training_step():
...
# Context manager — annotates a code block
with nvtx.annotate("data_loading", color="green"):
batch = next(dataloader)
- Do annotate training phases (forward, backward, optimizer, data loading) for nsys timeline clarity.
- Do not annotate for measurement — use CUDA events or manual timing instead.
- Do not over-annotate — too many fine-grained ranges add visual clutter and minor overhead.
For NVTX domains, categories, payloads, and legacy API details, see references/nvtx-api.md.
References
- references/benchmarking-patterns.md — CUDA Graph, CuTe DSL, Triton, Raw CUDA templates; warmup guidance; GPU hardware properties; reporting format
- references/nvtx-api.md — Domains, categories, payloads, legacy push/pop API
- references/pytorch-profiler-api.md — PyTorch 2.0+ profiler API changes (
device_time vs deprecated cuda_time)
1---2name: perf-workload-profiling3description: >4 Code instrumentation for timing workloads. Two scenarios:5 (1) Training loop — inject manual timing to report per-iteration latency,6 throughput (samples/sec), and data load time. (2) Standalone kernel/op —7 write CUDA event timing code with warmup, per-iteration statistics, and8 anti-pattern avoidance. Also covers NVTX annotation for labeling profiler9 timelines.10 NOT for: running or analyzing profiler tools (nsys, ncu, Nsight Systems,11 Nsight Compute), writing kernels (Triton, CuTe, CUDA), applying12 optimizations (CUDA Graphs, gradient checkpointing, fusion), or13 interpreting roofline/SOL% metrics.14 Triggers: "measure throughput", "benchmark this function", "time my15 training loop", "samples per second", "NVTX annotate", "instrument my16 dataloader", "data load time", "kernel timing", "how do I time".17license: Apache-2.018metadata:19 source: https://skillmd.com/skills/wenyi-li/perf-workload-profiling20 author: NVIDIA Corporation21---2223# Workload Profiling2425## Quick Reference2627Pick ONE path based on the workload type:2829| Workload | Approach | Section |30|----------|----------|---------|31| Training loop | Manual `torch.cuda.synchronize()` + `time.perf_counter()` with warmup | Loop Workloads — Manual Timing |32| Single kernel or op | Write CUDA event benchmark (pre-allocate, warmup, event pairs) | Non-Loop Workloads — CUDA Event Benchmarking |33| Add timeline labels for nsys | Use `@nvtx.annotate` decorator or context manager | NVTX Reference |3435## Principles3637- **Measure, don't guess.** Every performance claim must trace back to profiler output or structured measurement data. Never invent metrics.38- **Isolate steady-state.** Warmup costs (CUDA context init, cuDNN autotuning, JIT compilation) distort measurements. Always exclude warmup iterations before collecting data.39- **Use hardware timing.** CUDA events measure GPU time precisely. CPU timers (`time.perf_counter()`) include host overhead and miss asynchronous execution.40- **No sync inside measurement loops.** Each `torch.cuda.synchronize()` adds 10-50us overhead. Record CUDA events asynchronously, sync once at the end.41- **Pre-allocate everything.** Tensors, events, compiled kernels — all before the timing loop. For CuTe DSL kernels, pre-compile with `cute.compile()`.42- **Minimize profiler interference.** Start with lightweight measurement (manual timing for latency/throughput) and escalate to heavier tools (Kineto, nsys, ncu) only when lighter tools cannot answer the question.4344## Loop Workloads — Manual Timing4546For training loops and iterative workloads, use manual `torch.cuda.synchronize()` + `time.perf_counter()` timing with warmup to measure per-iteration latency, throughput, and data load time.4748### Injection Template4950Read the user's training script, understand the dataloader and loop structure, then inject timing code.5152```python53import time54import torch5556WARMUP = 557NUM_ITERS = 3058BATCH_SIZE = 128 # global batch size for throughput calculation5960iter_times = []61data_times = []6263for i, batch in enumerate(dataloader):64 if i >= WARMUP + NUM_ITERS:65 break6667 t_data_end = time.perf_counter()6869 torch.cuda.synchronize()70 t_start = time.perf_counter()7172 # ... existing training loop body ...7374 torch.cuda.synchronize()75 t_end = time.perf_counter()7677 if i >= WARMUP:78 iter_ms = (t_end - t_start) * 100079 iter_times.append(iter_ms)80 if i > 0:81 data_times.append((t_data_end - prev_iter_end) * 1000)82 print(f"[{i:04d}]: iter {iter_ms:.2f} ms, fps {BATCH_SIZE / (iter_ms / 1000):.2f}")8384 prev_iter_end = t_end8586import statistics87print(f"Average: iter {statistics.mean(iter_times):.2f} ms, "88 f"fps {BATCH_SIZE / (statistics.mean(iter_times) / 1000):.2f}")89```9091### Interpreting Results9293- **iter (ms)**: Wall-clock time per iteration (compute + communication, excluding data loading)94- **data (ms)**: Time spent in dataloader between iterations. If `data / iter > 0.2`, data loading is a bottleneck.95- **fps**: Global throughput in samples/second. Use with known FLOPs-per-sample to compute MFU.9697### Limitations9899Manual timing reports **aggregate** iteration timing — not per-sub-phase breakdown (forward, backward, optimizer). When the user asks **where time is spent within compute**:1001011. Add `torch.cuda.synchronize()` + `time.perf_counter()` around each sub-phase for a one-off diagnosis, OR1022. Add NVTX annotations and run with `nsys profile` for timeline visualization.103104## Non-Loop Workloads — CUDA Event Benchmarking105106For single kernels, one-shot inference, or standalone operations, write CUDA event benchmarking code directly.107108### PyTorch: Simple (Mean Only)109110```python111import torch112113def benchmark(fn, warmup=50, iters=100):114 for _ in range(warmup):115 fn()116 torch.cuda.synchronize()117118 start = torch.cuda.Event(enable_timing=True)119 end = torch.cuda.Event(enable_timing=True)120121 start.record()122 for _ in range(iters):123 fn()124 end.record()125 torch.cuda.synchronize()126127 return start.elapsed_time(end) / iters # ms per iteration128```129130### PyTorch: Detailed (Per-Iteration Stats)131132```python133import torch134import statistics135136def benchmark_detailed(fn, warmup=50, iters=100):137 for _ in range(warmup):138 fn()139 torch.cuda.synchronize()140141 starts = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]142 ends = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]143144 for i in range(iters):145 starts[i].record()146 fn()147 ends[i].record()148149 torch.cuda.synchronize()150 times = [starts[i].elapsed_time(ends[i]) for i in range(iters)]151152 return {153 "mean_ms": statistics.mean(times),154 "median_ms": statistics.median(times),155 "std_ms": statistics.stdev(times) if len(times) > 1 else 0,156 "min_ms": min(times),157 "max_ms": max(times),158 }159```160161### Anti-Patterns162163| Anti-Pattern | Problem |164|--------------|---------|165| `torch.cuda.synchronize()` before AND after each iteration | Adds ~10-50us overhead per iteration |166| `time.perf_counter()` for GPU timing | Measures CPU time, misses async GPU execution |167| Missing warmup | First iterations include JIT, clock ramp-up, context init |168| Allocating tensors inside measurement loop | Allocation overhead pollutes timing |169| Reporting only mean | Hides variance, outliers, bimodal distributions |170171For additional benchmarking templates (CUDA Graph, CuTe DSL, Triton, Raw CUDA), see [references/benchmarking-patterns.md](references/benchmarking-patterns.md).172173## NVTX Reference174175NVTX (NVIDIA Tools Extension) adds named annotations to profiler timelines. Use NVTX to label phases (forward, backward, optimizer) for readability in nsys — not for measurement.176177```python178import nvtx179180# Decorator — annotates every call181@nvtx.annotate("training_step", color="blue")182def training_step():183 ...184185# Context manager — annotates a code block186with nvtx.annotate("data_loading", color="green"):187 batch = next(dataloader)188```189190- **Do** annotate training phases (forward, backward, optimizer, data loading) for nsys timeline clarity.191- **Do not** annotate for measurement — use CUDA events or manual timing instead.192- **Do not** over-annotate — too many fine-grained ranges add visual clutter and minor overhead.193194For NVTX domains, categories, payloads, and legacy API details, see [references/nvtx-api.md](references/nvtx-api.md).195196## References197198- [references/benchmarking-patterns.md](references/benchmarking-patterns.md) — CUDA Graph, CuTe DSL, Triton, Raw CUDA templates; warmup guidance; GPU hardware properties; reporting format199- [references/nvtx-api.md](references/nvtx-api.md) — Domains, categories, payloads, legacy push/pop API200- [references/pytorch-profiler-api.md](references/pytorch-profiler-api.md) — PyTorch 2.0+ profiler API changes (`device_time` vs deprecated `cuda_time`)