Perf Host Optimization

Vendored from NVIDIA/TensorRT-LLM under Apache-2.0. Only the description was extended with broader inference-server triggers; the body and references are verbatim. NVIDIA's copyright + license header is preserved in this fork's LICENSE-Apache-2.0.txt.

references/hot-path-files.md is TRT-LLM-specific — it lists the most-edited Python files in TensorRT-LLM's serving stack. The methodology (find the top-K Python files responsible for inference-loop time, optimize them one at a time, validate via re-profile) carries over to vLLM / SGLang verbatim, but the file paths inside will not match those engines. Substitute the equivalents (e.g. vLLM's vllm/engine/*.py, vllm/core/scheduler.py) when applying this skill there.

Host Performance Optimization Skill

Automates detection and optimization of host-side (CPU) overhead in TensorRT-LLM's PyTorch backend.

When to Use

• GPU utilization is low during inference (CPU bottleneck suspected)
• User asks to reduce host overhead or CPU latency
• Optimizing PyExecutor throughput (requests/sec)
• Need line-by-line profiling of specific Python functions

Detecting a CPU Bottleneck

line_profiler measures where CPU time is spent but not whether CPU is the bottleneck. When user asks for confirmation, or there is no clear ending conditions for optimizations, use nsys (system-level trace) to confirm CPU is the limiting factor:

If nsys report is not available, use a rough heuristic: if doubling the batch size does not proportionally increase GPU utilization or throughput, CPU overhead is likely the bottleneck.

Using Analysis Skill Results

If the perf-host-analysis skill has already been run, use its output to skip the confirmation step and prioritize targets:

1. Detection verdict: If YES with host_prep_confirmed, start with _prepare_tp_inputs.
2. NVTX triage (from Root Cause): The top_regressing_ops in the handoff data block maps NVTX range names to source functions. Profile the function with the largest absolute delta first.
3. Cross-function triage: When the top NVTX regression is NOT in _prepare_tp_inputs (e.g., _fetch_new_requests, broadcast_requests, _update_requests), target that function's source file directly instead of defaulting to _prepare_tp_inputs. See references/trtllm-nvtx-ranges.md for the NVTX-to-source mapping.

Profiling Setup

line_profiler (Primary Method)

Environment Variables:

• TLLM_LINE_PROFILER_ENABLED=True — Enable the profiler
• TLLM_LINE_PROFILER_PATH — Output file path
• TLLM_LINE_PROFILER_FUNCTIONS — Additional functions to profile (comma-separated)

Function specification format:

# Class methods: module.path.ClassName.method_name
TLLM_LINE_PROFILER_FUNCTIONS="tensorrt_llm._torch.pyexecutor.model_engine.PyTorchModelEngine._prepare_tp_inputs"

# Standalone functions: module.path::function_name
TLLM_LINE_PROFILER_FUNCTIONS="tensorrt_llm._torch.pyexecutor.sampler::_group_requests_by_strategy_key"

# Multiple functions (comma-separated)
TLLM_LINE_PROFILER_FUNCTIONS="module.Class.method1,module.Class.method2"

CPU Affinity (Environment Factor)

CPU core affinity can significantly affect host overhead measurements, especially on multi-socket systems (e.g., B300). Pinning processes to cores near the GPU's NUMA node reduces cross-socket memory access latency.

• Check current affinity: taskset -p <pid> or numactl --show
• Pin to local NUMA node: numactl --cpunodebind=<node> --membind=<node>
• Impact: Up to 2x difference in host overhead on B300 systems

When comparing profiling results across runs, ensure CPU affinity is consistent. Do not externally modify the affinity, unless user requires to do this to examine the affects of this part. Document the affinity setting in each round's report if it varies.

Workspace & Suffix Management

Each profiling run should have a unique suffix to track progress across rounds:

EXTRA_SUFFIX=round0_baseline bash profile.sh
EXTRA_SUFFIX=round1_eliminate_redundant_iter bash profile.sh

Autonomous Optimization Loop

Run N rounds (default 3) of the following cycle:

FOR round = 1 to MAX_ROUNDS:

  1. PROFILE (with Drill-Down)
  2. ANALYZE (Multi-Option)
  3. OPTIMIZE (Apply Change)
  4. TEST (Unit Test Validation)
  5. VALIDATE (Re-Profile)
  6. REPORT

END FOR → FINAL SUMMARY

Phase 1: PROFILE (with Drill-Down)

• Run workload with profiler enabled
• Parse output: identify functions with highest Total time and lines with highest % Time
• CRITICAL: Drill down into sub-functions that are not yet profiled (see below)

Drill-Down Profiling

The default profiler covers top-level executor functions but not all sub-functions. When a profiled function shows most time in a single sub-call, you must drill down.

When: A single line consumes >80% of a function's time calling an unprofiled sub-function:

Line #      Hits         Time    Per Hit   % Time  Line Contents
==============================================================
  2848      4100  59200000000.0  14439024.4   98.7      output = self.model_engine.forward(...)

How:

1. Identify the sub-function's full qualified path (e.g., tensorrt_llm._torch.pyexecutor.model_engine.PyTorchModelEngine._prepare_tp_inputs)
2. Add it to TLLM_LINE_PROFILER_FUNCTIONS
3. Re-profile to get line-level data inside it
4. Now analyze the inner hotspots

For common drill-down targets, see references/hot-path-files.md.

Phase 2: ANALYZE (Multi-Option)

For the chosen hotspot:

1. Identify the top hotspots by absolute time (not just %) within the target function
2. Classify each hotspot by type. Summary table:

For detailed classification with code examples, see references/hotspot-classification.md.

3. Propose 2-4 optimization options in a table:

4. Select the best option and explain reasoning (prefer high-savings + low-risk)

For optimization patterns by type, see references/optimization-patterns.md.

Phase 3: OPTIMIZE (Apply Change)

• Apply the selected code change with Edit tool
• One optimization per round — keep changes minimal and targeted
• Record the exact change (file, line range, before/after) for potential rollback

Phase 4: TEST (Unit Test Validation)

Mandatory after each optimization. Find and run related UTs to verify correctness.

Finding related tests:

# Search by modified file name
grep -rl "model_engine\|PyTorchModelEngine" tests/unittest/_torch/executor/

# Search by modified function name
grep -rl "_prepare_tp_inputs\|prepare_inputs" tests/

Running tests:

# Run specific test file with stop-on-first-failure
pytest tests/unittest/_torch/executor/test_pytorch_model_engine.py -v -x --timeout=120

# Run specific test method
pytest tests/unittest/_torch/executor/test_pytorch_model_engine.py::PyTorchModelEngineTestCase::test_position_id_preparation -v -x

For the full UT-to-file mapping, see references/hot-path-files.md.

If tests fail:

1. Read the failure message
2. Rollback immediately (git checkout -- <file>)
3. Analyze why the optimization broke correctness
4. Try the next-best option from Phase 2

Phase 5: VALIDATE (Re-Profile)

• Re-run profiler with identical workload, using suffix round<N>_<description>
• Compare three things:
  1. Did the target hotspot time decrease?
  2. Did the overall function Total time decrease?
  3. Did benchmark metrics (TPOT, throughput) improve?

If regression detected (function time increased or metrics worsened):

• The "optimization" may have triggered a CPython pitfall — see references/optimization-patterns.md (CPython Pitfalls section)
• Rollback and try the next-best option from Phase 2

Phase 6: REPORT

Log for this round:

• Round number
• Hotspot location (file:line) and classification
• Optimization applied (with before/after code summary)
• Time delta: function Total time before → after
• Benchmark delta: TPOT, throughput before → after

Reading Profile Output

Timer unit: 1e-06 s
Total time: 1.234 s
File: /path/to/file.py
Function: my_function at line 100

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   100                                           def my_function(self):
   101       500    890000.0   1780.0     72.1       result = tensor.item()
   102       500    234567.0    469.1     19.0       return result

How to read effectively:

1. Start with Total time for each function — this is the overall budget
2. Sort lines mentally by absolute Time, not just % Time (3% of a 60s function = 1.8s)
3. Check Hits count to understand iteration patterns:
   • Hits = 2 × expected count → for x in range(1): loop overhead (2 hits = enter + exit check)
   • Hits ≫ expected → the line is inside a nested loop
4. Look for repeated patterns: if 10 lines each take 3% in a loop body, the loop itself costs 30%

Stopping Criteria

Stop the optimization loop when:

1. Iteration limit reached: Completed N rounds (default 3)
2. No actionable hotspots: Top hotspots are pure GPU compute (COMPUTE type)
3. Diminishing returns: < 5% improvement in last 2 rounds
4. Risk threshold: Further optimizations require architectural changes (e.g., Cython, struct-of-arrays)
5. Test failures: Cannot find an optimization that passes UTs

Primary success metric: Benchmark throughput (requests/sec or tokens/sec) as measured by the profiling script. line_profiler time reductions are leading indicators, but throughput is the ground truth — a function-level speedup that doesn't improve throughput is not a real win.

Final Summary Output

The final report should include:

• Rounds executed: Number of profile-optimize cycles completed
• Cumulative improvement: Total host time reduction (percentage and absolute)
• Benchmark metrics: Before/after comparison table (TPOT, throughput, ITL, E2EL)
• Optimizations applied: List of changes with file:line locations and classification
• Failed attempts: Any optimizations tried and reverted (with why)
• Remaining hotspots: Top bottlenecks that couldn't be optimized (with classification)
• Recommendations: Suggested follow-up for architectural changes if needed

For a concrete multi-round example, see references/examples.md.

Reference Files