Inference Decode Step Budget

Measure the low-concurrency (c=1..c=8) decode hot-path of a live vLLM pod FAST and CORRECTLY: where each token's time actually goes (GPU-busy vs host-idle gap vs comm), whether the workload is kernel-bound / host-bound / comm-bound, and what the addressable lever is. Uses vLLM's native /start_profile + /stop_profile HTTP endpoints (enable once, then restart-free shell-in captures in seconds). Encodes the correctness gates that prevent contaminated or under-counted budgets (clean single-stream driver, GPU-busy must include CUDA-graph execution, reconcile against driver-measured TPOT, reject load-time/lull/truncated captures). Triggers on "decode step budget", "c=1 hot path", "where does decode time go", "TPOT breakdown", "is decode kernel-bound or host-bound", "latency-tier profile", "GPU idle during decode", "start_profile", "torch profiler vllm", or any combination of "decode / c=1 / latency / TPOT / ITL" with "profile / budget / breakdown / hot-path / where" on a vllm deploy.

cfregly Updated 4 repo stars

File contents

cfregly/claude-perf-tune commit 6914632fe5

Frequently asked questions

npx get-skill add cfregly/inference-decode-step-budget-2