Nvidia Tensorrt LLM Perf Workload Profiling

Workload Profiling

Quick Reference

Pick ONE path based on the workload type:

Principles

• Measure, don't guess. Every performance claim must trace back to profiler output or structured measurement data. Never invent metrics.
• Isolate steady-state. Warmup costs (CUDA context init, cuDNN autotuning, JIT compilation) distort measurements. Always exclude warmup iterations before collecting data.
• Use hardware timing. CUDA events measure GPU time precisely. CPU timers (time.perf_counter()) include host overhead and miss asynchronous execution.
• No sync inside measurement loops. Each torch.cuda.synchronize() adds 10-50us overhead. Record CUDA events asynchronously, sync once at the end.
• Pre-allocate everything. Tensors, events, compiled kernels — all before the timing loop. For CuTe DSL kernels, pre-compile with cute.compile().
• Minimize profiler interference. Start with lightweight measurement (manual timing for latency/throughput) and escalate to heavier tools (Kineto, nsys, ncu) only when lighter tools cannot answer the question.

Loop Workloads — Manual Timing

For training loops and iterative workloads, use manual torch.cuda.synchronize() + time.perf_counter() timing with warmup to measure per-iteration latency, throughput, and data load time.

Injection Template

Read the user's training script, understand the dataloader and loop structure, then inject timing code.

import time
import torch

WARMUP = 5
NUM_ITERS = 30
BATCH_SIZE = 128  # global batch size for throughput calculation

iter_times = []
data_times = []

for i, batch in enumerate(dataloader):
    if i >= WARMUP + NUM_ITERS:
        break

    t_data_end = time.perf_counter()

    torch.cuda.synchronize()
    t_start = time.perf_counter()

    # ... existing training loop body ...

    torch.cuda.synchronize()
    t_end = time.perf_counter()

    if i >= WARMUP:
        iter_ms = (t_end - t_start) * 1000
        iter_times.append(iter_ms)
        if i > 0:
            data_times.append((t_data_end - prev_iter_end) * 1000)
        print(f"[{i:04d}]: iter {iter_ms:.2f} ms, fps {BATCH_SIZE / (iter_ms / 1000):.2f}")

    prev_iter_end = t_end

import statistics
print(f"Average: iter {statistics.mean(iter_times):.2f} ms, "
      f"fps {BATCH_SIZE / (statistics.mean(iter_times) / 1000):.2f}")

Interpreting Results

• iter (ms): Wall-clock time per iteration (compute + communication, excluding data loading)
• data (ms): Time spent in dataloader between iterations. If data / iter > 0.2, data loading is a bottleneck.
• fps: Global throughput in samples/second. Use with known FLOPs-per-sample to compute MFU.

Limitations

Manual timing reports aggregate iteration timing — not per-sub-phase breakdown (forward, backward, optimizer). When the user asks where time is spent within compute:

1. Add torch.cuda.synchronize() + time.perf_counter() around each sub-phase for a one-off diagnosis, OR
2. Add NVTX annotations and run with nsys profile for timeline visualization.

Non-Loop Workloads — CUDA Event Benchmarking

For single kernels, one-shot inference, or standalone operations, write CUDA event benchmarking code directly.

PyTorch: Simple (Mean Only)

import torch

def benchmark(fn, warmup=50, iters=100):
    for _ in range(warmup):
        fn()
    torch.cuda.synchronize()

    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)

    start.record()
    for _ in range(iters):
        fn()
    end.record()
    torch.cuda.synchronize()

    return start.elapsed_time(end) / iters  # ms per iteration

PyTorch: Detailed (Per-Iteration Stats)

import torch
import statistics

def benchmark_detailed(fn, warmup=50, iters=100):
    for _ in range(warmup):
        fn()
    torch.cuda.synchronize()

    starts = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]
    ends = [torch.cuda.Event(enable_timing=True) for _ in range(iters)]

    for i in range(iters):
        starts[i].record()
        fn()
        ends[i].record()

    torch.cuda.synchronize()
    times = [starts[i].elapsed_time(ends[i]) for i in range(iters)]

    return {
        "mean_ms": statistics.mean(times),
        "median_ms": statistics.median(times),
        "std_ms": statistics.stdev(times) if len(times) > 1 else 0,
        "min_ms": min(times),
        "max_ms": max(times),
    }

Anti-Patterns

For additional benchmarking templates (CUDA Graph, CuTe DSL, Triton, Raw CUDA), see references/benchmarking-patterns.md.

NVTX Reference

NVTX (NVIDIA Tools Extension) adds named annotations to profiler timelines. Use NVTX to label phases (forward, backward, optimizer) for readability in nsys — not for measurement.

import nvtx

# Decorator — annotates every call
@nvtx.annotate("training_step", color="blue")
def training_step():
    ...

# Context manager — annotates a code block
with nvtx.annotate("data_loading", color="green"):
    batch = next(dataloader)

• Do annotate training phases (forward, backward, optimizer, data loading) for nsys timeline clarity.
• Do not annotate for measurement — use CUDA events or manual timing instead.
• Do not over-annotate — too many fine-grained ranges add visual clutter and minor overhead.

For NVTX domains, categories, payloads, and legacy API details, see references/nvtx-api.md.

References

• references/benchmarking-patterns.md — CUDA Graph, CuTe DSL, Triton, Raw CUDA templates; warmup guidance; GPU hardware properties; reporting format
• references/nvtx-api.md — Domains, categories, payloads, legacy push/pop API
• references/pytorch-profiler-api.md — PyTorch 2.0+ profiler API changes (device_time vs deprecated cuda_time)