Profile vLLM Performance
Find the dominant loss mechanism, prove it with a correlated timeline, make the
smallest causal change, and validate the result with a matched A/B run. A capacity
number or average GPU-utilization sample is not bubble attribution.
Freeze The Comparison
Record the exact code, container, model revision, precision, hardware, scheduler
settings, workload shape, media, prompt, token budget, sampling, cache state,
concurrency, and success criteria. Prove semantic correctness before optimizing.
Do not compare runtimes when any of these dimensions differ unless that dimension is
the declared variable.
Establish The Signal
- Verify actual output-token counts, successful work, fresh-stream coverage,
dropped chunks, cancellations, and cleanup. Early EOS or silent failure is not
throughput.
- Reproduce at fixed load below the knee, near the highest stable point, and just
beyond it when practical. Keep capacity search separate from profiling.
- Record TTFT, inter-token latency, throughput, queue delay, batch size, scheduled
tokens, active sequences, preemptions, KV occupancy, GPU memory, and phase
durations available in the checked-out version.
- Use scenario-scoped DCGM or
nvidia-smi dmon for the resource envelope and a
short Nsight Systems or repository profiler capture for attribution. Profilers
perturb timing; never use a profiled run as the capacity result.
Define a performance bubble as GPU-idle or low-occupancy time while eligible work is
queued. Idle time caused by insufficient offered load is not a runtime bubble.
If the scheduler queue is empty throughout every observed idle interval, first check
whether eligible work is waiting upstream in media arrival, decode, preprocessing, or
engine submission. Upstream waiting work is frontend or media starvation, not an
offered-load gap. When neither the scheduler nor upstream stages have eligible work
waiting, classify the result as insufficient offered load and make the first experiment
a higher-fixed-concurrency replay with every other workload dimension unchanged. Do
not recommend scheduler, kernel, batching, cache, prefill, or decode optimization until
that replay shows GPU idle while eligible work remains queued.
Correlate The Timeline
Use stable request, sequence, stream, and chunk identifiers across:
- media arrival and decode;
- CPU frame selection, resize, normalization, tokenization, and processor work;
- host-to-device transfer and synchronization;
- vision encoder and projector;
- text and visual prefill;
- scheduler admission, batch formation, KV allocation, preemption, and dispatch;
- decode, sampling, detokenization, and output delivery.
Capture only enough NVTX or structured timing to align these phases with CUDA
kernels, memory copies, CPU wakeups, and scheduler state. Avoid high-volume logging
in the measured path.
Read references/bubble-playbook.md for the evidence
matrix and the shortest discriminating experiment for each bubble class.
Prove The Dominant Bubble
For each candidate, state the precise idle or under-occupancy interval, whether
eligible work was queued, the event immediately preceding it, its frequency and
share of measured time, and the observation that rules out the nearest competing
explanation. Rank candidates by recoverable critical-path time, not visual prominence
in one trace.
Do not infer a decode bottleneck from an OSL capacity gap alone. Compare matched
fixed-concurrency prefill and decode timelines plus tokens per scheduler step.
For serialized vision, first compare an existing batched preprocessing or vision-
encoding path using the same fixed visual shapes and request set. State its acceptance
criteria explicitly: require output parity and multimodal visual-token count, order,
and position parity before attributing any performance improvement to the change.
Fix And Validate
Stop at the first rung that removes the proven bubble:
- Correct benchmark or configuration defects such as early EOS, mismatched shapes,
cache policy, scheduler limits, or insufficient offered load.
- Tune an existing runtime knob supported by the checked-out version.
- Reuse an existing batching, preprocessing, cache, CUDA-graph, tensor-IPC, or
scheduler path that is configured incorrectly or bypassed.
- Fix an integration boundary that serializes or copies work unnecessarily.
- Change scheduling or model-runner code only when traces prove the existing path
cannot express the required overlap.
- Change a kernel only after ruling out launch, scheduling, shape, and data movement.
Change one causal variable at a time. Repeat the short trace, then run an unprofiled
matched A/B or A/B/B/A comparison at fixed concurrency. Rerun the capacity boundary
only when capacity is the claim. Preserve output parity, ordering, cancellation, KV
lifetime, and multimodal token and position invariants.
Report confirmed, improved, inconclusive, or invalid comparison, followed by
the dominant bubble, recoverable-time estimate, evidence, ruled-out alternatives,
fix, before/after distributions, artifact paths, remaining bottleneck, and smallest
next experiment.
Do not install profilers, launch costly GPU work, push changes, or mutate a shared
machine without authorization.
1---2name: profile-vllm-performance3description: Profile vLLM and RT-VLM inference to identify and remove GPU-idle gaps, underfilled batches, transfer stalls, serialized multimodal work, scheduler gaps, or KV pressure. Use this skill when GPU utilization is unexpectedly low or bursty, capacity trails another runtime, TTFT or ITL regresses, or multimodal prefill and decode appear serialized. Not for running a first benchmark without a reproducible fixed-load signal.4license: Apache-2.05metadata:6 source: https://skillmd.com/skills/nvidia-ai-blueprints/profile-vllm-performance7 version: "3.3.0-rc0"8 requires-vss: ">=3.2.0,<4.0.0"9 author: "NVIDIA Video Search and Summarization Team"10 github-url: "https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization"11 tags: "nvidia rt-vlm vllm performance profiling gpu"12---1314# Profile vLLM Performance1516Find the dominant loss mechanism, prove it with a correlated timeline, make the17smallest causal change, and validate the result with a matched A/B run. A capacity18number or average GPU-utilization sample is not bubble attribution.1920## Freeze The Comparison2122Record the exact code, container, model revision, precision, hardware, scheduler23settings, workload shape, media, prompt, token budget, sampling, cache state,24concurrency, and success criteria. Prove semantic correctness before optimizing.25Do not compare runtimes when any of these dimensions differ unless that dimension is26the declared variable.2728## Establish The Signal29301. Verify actual output-token counts, successful work, fresh-stream coverage,31 dropped chunks, cancellations, and cleanup. Early EOS or silent failure is not32 throughput.332. Reproduce at fixed load below the knee, near the highest stable point, and just34 beyond it when practical. Keep capacity search separate from profiling.353. Record TTFT, inter-token latency, throughput, queue delay, batch size, scheduled36 tokens, active sequences, preemptions, KV occupancy, GPU memory, and phase37 durations available in the checked-out version.384. Use scenario-scoped DCGM or `nvidia-smi dmon` for the resource envelope and a39 short Nsight Systems or repository profiler capture for attribution. Profilers40 perturb timing; never use a profiled run as the capacity result.4142Define a performance bubble as GPU-idle or low-occupancy time while eligible work is43queued. Idle time caused by insufficient offered load is not a runtime bubble.44If the scheduler queue is empty throughout every observed idle interval, first check45whether eligible work is waiting upstream in media arrival, decode, preprocessing, or46engine submission. Upstream waiting work is frontend or media starvation, not an47offered-load gap. When neither the scheduler nor upstream stages have eligible work48waiting, classify the result as insufficient offered load and make the first experiment49a higher-fixed-concurrency replay with every other workload dimension unchanged. Do50not recommend scheduler, kernel, batching, cache, prefill, or decode optimization until51that replay shows GPU idle while eligible work remains queued.5253## Correlate The Timeline5455Use stable request, sequence, stream, and chunk identifiers across:5657- media arrival and decode;58- CPU frame selection, resize, normalization, tokenization, and processor work;59- host-to-device transfer and synchronization;60- vision encoder and projector;61- text and visual prefill;62- scheduler admission, batch formation, KV allocation, preemption, and dispatch;63- decode, sampling, detokenization, and output delivery.6465Capture only enough NVTX or structured timing to align these phases with CUDA66kernels, memory copies, CPU wakeups, and scheduler state. Avoid high-volume logging67in the measured path.6869Read [references/bubble-playbook.md](references/bubble-playbook.md) for the evidence70matrix and the shortest discriminating experiment for each bubble class.7172## Prove The Dominant Bubble7374For each candidate, state the precise idle or under-occupancy interval, whether75eligible work was queued, the event immediately preceding it, its frequency and76share of measured time, and the observation that rules out the nearest competing77explanation. Rank candidates by recoverable critical-path time, not visual prominence78in one trace.7980Do not infer a decode bottleneck from an OSL capacity gap alone. Compare matched81fixed-concurrency prefill and decode timelines plus tokens per scheduler step.8283For serialized vision, first compare an existing batched preprocessing or vision-84encoding path using the same fixed visual shapes and request set. State its acceptance85criteria explicitly: require output parity and multimodal visual-token count, order,86and position parity before attributing any performance improvement to the change.8788## Fix And Validate8990Stop at the first rung that removes the proven bubble:91921. Correct benchmark or configuration defects such as early EOS, mismatched shapes,93 cache policy, scheduler limits, or insufficient offered load.942. Tune an existing runtime knob supported by the checked-out version.953. Reuse an existing batching, preprocessing, cache, CUDA-graph, tensor-IPC, or96 scheduler path that is configured incorrectly or bypassed.974. Fix an integration boundary that serializes or copies work unnecessarily.985. Change scheduling or model-runner code only when traces prove the existing path99 cannot express the required overlap.1006. Change a kernel only after ruling out launch, scheduling, shape, and data movement.101102Change one causal variable at a time. Repeat the short trace, then run an unprofiled103matched A/B or A/B/B/A comparison at fixed concurrency. Rerun the capacity boundary104only when capacity is the claim. Preserve output parity, ordering, cancellation, KV105lifetime, and multimodal token and position invariants.106107Report `confirmed`, `improved`, `inconclusive`, or `invalid comparison`, followed by108the dominant bubble, recoverable-time estimate, evidence, ruled-out alternatives,109fix, before/after distributions, artifact paths, remaining bottleneck, and smallest110next experiment.111112Do not install profilers, launch costly GPU work, push changes, or mutate a shared113machine without authorization.