Perf Host Analysis

Vendored from NVIDIA/TensorRT-LLM under Apache-2.0. Only the description was extended with broader inference-server triggers; the body, scripts, and references are verbatim. NVIDIA's copyright + license header is preserved in this fork's LICENSE-Apache-2.0.txt.

references/trtllm-nvtx-ranges.md is TRT-LLM-specific (it lists the NVTX range names emitted by TensorRT-LLM's PyTorch backend). It is still useful as a worked example of how a serving engine annotates iterations, but the exact range names will not match vLLM / SGLang. The Phase-2 methodology (compare per-NVTX-range latency across versions to find the regression) is engine-agnostic — only the specific names differ.

Host Performance Analysis

Analyze host/CPU overhead in TensorRT-LLM inference workloads from nsys traces. This skill operates in two phases:

When to Use

• Before starting host optimization work — confirms the bottleneck is real (Detection)
• As a sub-step of perf-analysis for bottleneck classification (Detection)
• When GPU utilization is suspiciously low and you need to know why (Detection)
• When throughput regressed but GPU kernel execution times are unchanged (Root Cause)
• When the gap between forward step iterations has increased (Root Cause)
• To compare inter-iteration overhead between two versions of the inference engine (Root Cause)

Do NOT use when:

• The regression is in individual kernel performance (use perf-nsight-compute-analysis)
• You need to profile a workload from scratch (use workload-instrumentation first)
• The issue is NCCL communication (use distributed analysis)

Prerequisites

• An nsys trace file (.sqlite or .nsys-rep) from a TRT-LLM benchmark run
• For Root Cause comparison: two traces (baseline and target)
• Python 3 with sqlite3 support

Key Concepts

Host Overhead in LLM Inference

In an LLM inference loop, each iteration consists of:

[inter-step gap] -> [_forward_step] -> [inter-step gap] -> [_forward_step] -> ...

The forward step includes GPU kernel execution (GEMM, attention, normalization, allreduce) plus host-side preparation (_prepare_inputs, resource allocation).

The inter-step gap includes all host-side work between forward steps:

• Request scheduling (_schedule)
• Request fetching (_fetch_new_requests)
• Request broadcasting (broadcast_requests — TP configurations)
• Sampling (_sample_async)
• Request processing (_process_requests)
• Response handling (_handle_responses)
• Request state updates (_update_requests)

Hidden vs Exposed Host Overhead

Host overhead only hurts performance when it is exposed — the GPU is idle, waiting for the host to submit work. When host prep runs while the GPU is still busy with previous kernels, it is hidden and costs nothing in wall-clock time.

Scenario A — Host prep hidden (GPU-bound, healthy)

time ------>
Host: |prep N|launch|...wait...|post|prep N+1|launch|...wait...|post|
GPU:         |========= kernels N =========|======= kernels N+1 ======|
              GPU always busy; host prep is hidden behind GPU execution
              exposed host overhead = 0

Scenario B — Host prep exposed (host-bound, bottleneck)

time ------>
Host: |prep N|launch|post|======= long prep N+1 =======|launch|post|
GPU:         |kernels N|     ^^^^ GPU IDLE ^^^^         |kernels N+1|
                              exposed host overhead
                              (directly adds to wall time)

Scenario C — Partially hidden (common in practice)

time ------>
Host: |prep N|launch|post|======== prep N+1 ========|launch|post|
GPU:         |======== kernels N ========|   ^IDLE^  |kernels N+1|
              hidden portion ----------->   exposed
              (overlaps GPU, free)          (GPU idle, costs wall time)

Forward Step Isolation via Allreduce Pattern

In tensor-parallel configurations, each forward step executes a fixed number of allreduce operations (one per transformer layer communication point).

The algorithm:

1. Find all allreduce_fusion kernels in the trace
2. Group consecutive allreduce kernels separated by < 1ms into "iterations"
3. Identify the most common iteration size (= kernels per forward step)
4. Detect phase boundaries where consecutive iterations are separated by > 100ms
5. Select the last phase with the common iteration size as the benchmark phase

See references/iteration-isolation-techniques.md for full details including NVTX-based and kernel-density approaches.

Phase Classification (Context vs Generation)

TRT-LLM iterations are classified by NVTX marker text into context (eager, no CUDA graphs) and generation (CUDA graph replay). Aggregate metrics can mask phase-specific bottlenecks, so per-phase analysis is critical.

NVTX marker format: [Executor] _forward_step {iter}: {N} ctx reqs, {M} gen reqs

See references/phase-classification.md for extraction code and per-phase aggregation.

Phase 1: Detection (YES/NO Verdict)

Determine whether host overhead is the primary bottleneck for a TRT-LLM workload.

Detection Metrics

Six metrics, grouped into four categories. See references/metrics.md for full definitions and SQL queries.

Host prep confirmation rule: Host prep is a confirmed bottleneck only when both M3b AND M3c cross their thresholds.

Thresholds are configurable with per-phase variants. See references/thresholds.md.

Detection Workflow

Step 1: Input Validation

# Accept .sqlite or .nsys-rep
ls -la <trace_file>

# If .nsys-rep, export to SQLite first
nsys export -t sqlite -o <output.sqlite> <input.nsys-rep>

Step 2: Extract Metrics via SQL

All Detection metrics (M1, M2, M4, M5) are computed directly from the nsys SQLite trace using SQL queries. No external tools are required.

-- M1: GPU idle ratio + M4: GPU utilization
-- Step A: Get analysis window
SELECT MIN(start) AS window_start, MAX(end) AS window_end,
       (MAX(end) - MIN(start)) / 1000.0 AS total_time_us
FROM CUPTI_ACTIVITY_KIND_KERNEL;

-- Step B: Compute GPU active time (merge overlapping kernel ranges)
-- Export kernel start/end pairs and merge in Python, or use the
-- approximate sum (accurate when kernel overlap is minimal):
SELECT SUM(end - start) / 1000.0 AS approx_gpu_active_us
FROM CUPTI_ACTIVITY_KIND_KERNEL;
-- gpu_idle_us ≈ total_time_us - gpu_active_us
-- gpu_idle_ratio = gpu_idle_us / total_time_us  (M1, threshold >0.30)
-- gpu_utilization = 1 - gpu_idle_ratio           (M4, threshold <0.60)

-- M2: Launch overhead ratio
SELECT SUM(r.end - r.start) / 1000.0 AS cudaLaunchKernel_us
FROM CUPTI_ACTIVITY_KIND_RUNTIME r
JOIN StringIds s ON r.nameId = s.id
WHERE s.value = 'cudaLaunchKernel';
-- launch_overhead_ratio = cudaLaunchKernel_us / total_time_us  (threshold >0.10)

-- M5: NCCL ratio (caveat metric)
SELECT SUM(k.end - k.start) / 1000.0 AS nccl_us
FROM CUPTI_ACTIVITY_KIND_KERNEL k
JOIN StringIds s ON k.shortName = s.id
WHERE s.value LIKE '%nccl%';
-- nccl_ratio = nccl_us / gpu_active_us  (threshold >0.20)

For precise GPU active time (merging overlapping ranges), export kernel (start, end) pairs and merge in Python — see references/metrics.md for the full approach.

Step 3: Extract Per-Iteration Metrics

Parse _forward_step NVTX ranges and classify iterations into context/generation phases. See references/phase-classification.md.

Step 4: Compute M3 (Optional — Advanced)

M3 (Host Prep Exposed Ratio) requires intersecting NVTX host-prep ranges with GPU idle gaps — a range-intersection computation that is not suitable for inline SQL. This metric is optional for the Detection verdict; M1+M2+M4+M5 are usually sufficient.

If M3 is needed, compute it in Python:

1. Export merged GPU idle gaps (from Step 2 range merging)
2. Export NVTX ranges matching the host-prep marker (see references/metrics.md for the configurable range name)
3. Intersect each NVTX range with the idle gaps to compute exposed_us
4. Apply M3a/M3b/M3c formulas from references/metrics.md

Step 5: Compute All Metrics and Apply Decision Logic

Aggregate Verdict:

# Core metrics (always available from SQL):
core_crossed = count of [M1, M2, M4] that cross their threshold

# Optional M3 metrics (if computed):
if M3 available:
    crossed_count = core_crossed + count of [M3a, M3b, M3c] that cross
    applicable_count = 6
else:
    crossed_count = core_crossed
    applicable_count = 3

if crossed_count >= 2:
    aggregate_verdict = YES

if M3 available AND M3b > threshold AND M3c > threshold:
    host_prep_confirmed = true

if nccl_ratio > NCCL_RATIO_CAVEAT_THRESHOLD:
    add caveat

Per-Phase Verdicts: Apply phase-specific thresholds to context and generation iterations separately.

Overall Verdict:

if aggregate_verdict == YES or context_verdict == YES or generation_verdict == YES:
    overall_verdict = YES
else:
    overall_verdict = NO

Per-phase analysis can elevate the verdict but never demote it.

Step 6: Generate Report

Format using the template in references/output-format.md.

Next Steps:

• If YES -> Proceed to Phase 2 (Root Cause) below, then use perf-host-optimization skill
• If NO -> Use perf-nsight-compute-analysis for kernel SOL% or trace-interpretation for full classification

Phase 2: Root Cause Analysis

Identify which specific host operations regressed and by how much. Works with a single trace (breakdown) or two traces (comparison).

Principles

1. Isolate forward steps, not the full trace. nsys traces contain warmup, JIT compilation, model loading, and teardown. Only forward step iterations represent actual inference performance.

2. Use structural kernel patterns for iteration detection. Allreduce kernel grouping is more robust than kernel density or time-window heuristics.

3. Compare steady-state iterations. Filter to iterations with identical workload (same batch size, same ctx/gen mix) for clean comparison.

4. Per-step metrics, not totals. When benchmark windows differ in duration or step count, always compare per-step averages.

Root Cause Workflow

Step 1: Collect nsys Traces

Profile both versions (if comparing) with identical settings:

nsys profile -o /path/to/trace \
  -t cuda,nvtx,osrt \
  --force-overwrite=true \
  --cuda-memory-usage=true \
  -w true \
  <benchmark_command> --num_requests 500

Step 2: Export to SQLite

nsys export --type=sqlite --force-overwrite=true -o trace.sqlite trace.nsys-rep

Step 3: Run Host Overhead Analysis

# Two-trace comparison
python scripts/analyze_host_overhead.py \
  --baseline /path/to/baseline/trace.sqlite \
  --target /path/to/target/trace.sqlite \
  --baseline-label "v1.1" \
  --target-label "main" \
  --output /path/to/output/analysis.txt

# Single-trace breakdown
python scripts/analyze_host_overhead.py \
  --baseline /path/to/trace.sqlite \
  --baseline-label "current"

Step 4: Interpret Results

The script produces:

1. Allreduce-based iteration detection — confirms forward step boundaries
2. Per-step wall time comparison — quantifies the regression
3. NVTX per-step breakdown — identifies which host operations regressed
4. GPU kernel comparison — confirms GPU execution is unchanged
5. CUDA API comparison — detects kernel launch overhead changes

Reading the Output

Per-Step Wall Time:

Avg wall time per step: 3,317 us (baseline) vs 3,978 us (target)  +19.9%

This is the primary regression metric.

NVTX Breakdown:

Operation           | baseline (us/step) | target (us/step) | Delta    | Status
_fetch_new_requests |               36   |             270  | +234     | REGRESSION
broadcast_requests  |                -   |             250  | +250     | NEW
_update_requests    |              413   |             723  | +310     | REGRESSION
_sample_async       |            1,163   |             720  | -443     | IMPROVED
_process_requests   |            1,056   |             390  | -666     | IMPROVED

Focus on operations with large absolute deltas. Check whether improvements offset regressions.

GPU Kernel Comparison:

Kernels per step (launched): 6.2 (baseline) vs 21.9 (target)  +253%

More individual launches = more host-side launch overhead.

Common Patterns and Root Causes

Pattern 1: Request Management Refactor

Symptom: _fetch_new_requests regressed 5-10x, new broadcast_requests operation. Cause: Request fetching refactored to support multi-rank broadcasting in TP. Impact: +500-1000 us/step in TP configurations. Mitigation: Optimize broadcast path; batch request state updates.

Pattern 2: Increased Kernel Launch Count

Symptom: 3-5x more cudaLaunchKernel calls per step, similar GPU time. Cause: Operations that were fused or graph-captured are now individual launches. Impact: +50-100 us/step from launch overhead alone. Mitigation: Re-fuse kernels; extend CUDA graph capture scope.

Pattern 3: New Bookkeeping Operations

Symptom: New NVTX ranges like _write_finish_reasons, handle_additional_outputs. Cause: New features added to the inference loop without overhead budgeting. Impact: +100-200 us/step each. Mitigation: Defer non-critical bookkeeping to async paths; batch updates.

Pattern 4: Flashinfer JIT Warmup Masquerading as Inference

Symptom: Massive elementwise/reduce kernel counts in "steady state" analysis. Cause: Analysis window includes flashinfer JIT compilation phase. Impact: False positive — not a real regression. Detection: Forward steps identified by allreduce pattern do NOT contain these kernels. Fix: Use allreduce-based iteration isolation, not kernel density or time windows.

Pattern 5: Context-Only Bottleneck (Masked by Aggregate)

Symptom: Aggregate metrics below threshold (e.g., GPU idle 25%), but context iterations have 50% GPU idle. Cause: Generation iterations (healthy, CUDA graphs) dilute the context-phase bottleneck. Detection: Per-phase analysis in Detection phase catches this. Fix: Optimize context-phase host preparation (_prepare_tp_inputs eager path).

Pitfalls

1. shortName is an Integer ID

In CUPTI_ACTIVITY_KIND_KERNEL, shortName is an integer referencing StringIds.id. Always join:

SELECT s.value, COUNT(*), SUM(k.end - k.start)/1000.0
FROM CUPTI_ACTIVITY_KIND_KERNEL k
JOIN StringIds s ON k.shortName = s.id
WHERE k.start >= ? AND k.start < ?
GROUP BY s.value ORDER BY 3 DESC

2. NVTX textId vs text

Most NVTX events have textId (integer) but NULL text. Join with StringIds:

SELECT s.value, n.start, n.end
FROM NVTX_EVENTS n
JOIN StringIds s ON n.textId = s.id
WHERE s.value LIKE '%_forward_step%'

3. Duplicate NVTX Ranges from TP Ranks

In TP configurations, each rank reports NVTX ranges independently. De-duplicate by grouping entries within 100us of each other.

4. Negative Inter-Step Gaps

When TP ranks report overlapping NVTX ranges, gap = next_start - prev_end can be negative. Use the maximum end time when de-duplicating.

5. Benchmark Window Selection

The allreduce-based window captures context+generation phases; steady-state NVTX filtering captures generation-only. Both are valid; use the appropriate one for your comparison goal.

nsys SQLite Schema Reference

Key Tables

Useful Queries

Find all NVTX range names:

SELECT DISTINCT s.value, COUNT(*)
FROM NVTX_EVENTS n
JOIN StringIds s ON n.textId = s.id
WHERE n.end > 0
GROUP BY s.value
ORDER BY COUNT(*) DESC

Allreduce kernels timeline:

SELECT (k.start - (SELECT MIN(start) FROM CUPTI_ACTIVITY_KIND_KERNEL))/1e9 AS t_sec,
       (k.end - k.start)/1000.0 AS dur_us
FROM CUPTI_ACTIVITY_KIND_KERNEL k
JOIN StringIds s ON k.shortName = s.id
WHERE s.value LIKE '%allreduce%'
ORDER BY k.start

CUDA API breakdown during a time window:

SELECT s.value, COUNT(*), SUM(r.end - r.start)/1000.0 AS total_us
FROM CUPTI_ACTIVITY_KIND_RUNTIME r
JOIN StringIds s ON r.nameId = s.id
WHERE r.start >= ? AND r.start < ?
GROUP BY s.value ORDER BY total_us DESC

Case Study: Llama 3.2 1B TP=2 Regression (v1.1 -> main, Feb 2025)

Symptom

19.2% throughput regression: 445.63 -> 360.02 req/sec.

False Starts

1. Whole-trace analysis attributed regression to massive elementwise/reduce kernels -> flashinfer JIT, not inference.
2. CUDA graph analysis suggested graph usage changed -> both versions use similar patterns.
3. Kernel density windowing selected wrong time window -> captured JIT phase.

Correct Analysis (Detection + Root Cause)

Detection confirmed host overhead is the bottleneck (GPU idle ratio high, GPU kernels unchanged).

Root Cause via allreduce + NVTX analysis:

1. Allreduce_fusion grouping (66 per iteration) isolated forward steps
2. GPU kernel profiles were identical between versions
3. Per-step wall time: 3,317 us -> 3,978 us (+20%)
4. Inter-step gap P50: 2,543 us -> 4,468 us (+76%)

Root Cause Breakdown

Lesson

Regression was entirely in the request management layer between forward steps. GPU computation was unchanged. Structural iteration isolation and steady-state NVTX comparison were essential for the correct root cause.

Handoff to Optimization

When analysis is complete and the verdict is YES, hand off to the perf-host-optimization skill with the following context:

1. Detection verdict and evidence: Which metrics crossed thresholds (M1-M5), whether host prep was confirmed as the bottleneck (M3b+M3c), and per-phase breakdown.

2. NVTX-based triage (from Root Cause output): The top regressing NVTX operations by absolute delta (us/step) guide which function to profile first with line_profiler. Map NVTX range names to source functions:

   • _prepare_tp_inputs → PyTorchModelEngine._prepare_tp_inputs
   • _fetch_new_requests / broadcast_requests → request management in PyExecutor
   • _update_requests / _process_requests → request lifecycle in PyExecutor
   • _sample_async → sampler pipeline

3. Handoff data block: Include the structured data from references/output-format.md (see "Handoff to Optimization" section) so the optimization skill can prioritize without re-running analysis.

Reference