Inference Kernel Profile

inference-kernel-profile

Purpose

Capture per-kernel CUDA timeline + absolute-duration profile data from a live vLLM inference pod without rebuilding the production image. Uses kubectl debug --share-processes to attach an nsys-enabled sidecar to an existing pod. The sidecar profiles the running vllm engine PID and writes .nsys-rep + summary.csv into a sidecar emptyDir that the operator extracts via kubectl cp.

What nsys is for vs the sibling skills

Fast path for the decode/latency tier (prefer over a restart): if the question is "where does c=1 decode time go", use inference-decode-step-budget - it drives vLLM's /start_profile + /stop_profile HTTP endpoints so you capture restart-free in seconds, and it bakes in the correctness gates (clean single-stream driver. GPU-busy must include CUDA-graph execution. Reconcile against driver TPOT). This skill (nsys sidecar/timeline) is for prefill / throughput-tier hot-spots and the deep CUDA-graph timeline.

Note on zymtrace symbol resolution: zymtrace DOES resolve SASS-level kernel symbols - the per-kernel join returns fully-resolved kernel names directly from zymtrace ClickHouse via interp_funcs. A high "native unresolved" share usually means the query joined wrong (filtered out the kernel-name match), not a tooling gap. Use zymtrace-anchored-query first for "what kernels run" - it's free + no cluster mutation. Use nsys for the things zymtrace CAN'T give you (timeline, absolute ns, NVTX, cuda graph events).

When to use

• Investigating a specific bottleneck where the operator already ran a perf-bench sweep + zymtrace and wants per-kernel detail beyond the category breakdown.
• A/B'ing two helm configs to see which kernels shifted.
• One-shot release-readiness check before a major deploy change.

Do not use this skill for:

• Continuous always-on capture - that's what zymtrace is for. nsys is on-demand only because it adds ~5% per-pod overhead during capture.
• Multi-pod fleet-level profiling - this skill targets ONE pod at a time. For fleet capture use the zymtrace profiler DaemonSet.

Before attaching: bound the question

Apply the Dean-Ghemawat performance-hints adaptation before paying the capture overhead. Name the end-to-end metric, the suspected contributor, its current profile share if known, and the maximum win possible if it were removed. For c=1 decode, route to inference-decode-step-budget first. For a known hot kernel that needs stalls or roofline data, route directly to inference-kernel-ncu-profile.

If no profile share exists, this nsys capture may establish it. Record that as the question. Do not start with an untargeted "capture everything" request.

Sidecar image

Two paths:

• Path A (recommended): pre-built ghcr.io/cfregly/nsys-sidecar:0.1.0@sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d (publicly readable. Manifest digest sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d).
• Path B (zero-build): nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04 with apt install nsight-systems-cli at attach time

Recipe

1. Identify the target vllm pod

NS=inference
TARGET_POD=$(kubectl -n $NS get pods -l app=basic-inference -o jsonpath='{.items[0].metadata.name}')
echo "Target pod: $TARGET_POD"

2. Attach the sidecar (one-shot, non-disruptive)

SIDECAR=nsys-debug-$(date +%s)
kubectl -n $NS debug $TARGET_POD \
  --image=ghcr.io/cfregly/nsys-sidecar:0.1.0@sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d \
  --container=$SIDECAR \
  --share-processes --target=basic-inference \
  -- sleep 3600

--share-processes is the critical flag: it puts the sidecar in the target pod's PID namespace so nsys profile --attach-pid <vllm-pid> can see the engine process.

3. Capture (120s default duration)

VLLM_PID=$(kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- \
  pgrep -f "vllm serve" | head -1)

kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c "
  nsys profile \
    --output=/profiling/capture \
    --capture-range=cudaProfilerApi \
    --duration=120 \
    --force-overwrite=true \
    --attach-pid=$VLLM_PID \
    --sample=cpu --trace=cuda,nvtx,osrt \
    --cuda-graph-trace=node \
    --sampling-frequency=1000
"

Gotcha 1 - CUDA graphs (must-read). vLLM runs decode (and often prefill) inside CUDA graphs. Without --cuda-graph-trace=node the graph executes as one opaque GRAPH_TRACE span and the constituent kernels do NOT appear in cuda_gpu_kern_sum - so a KERNEL-only "GPU-busy" sum under-counts real busy time and a kernel ranking misses the in-graph forward kernels. Always pass --cuda-graph-trace=node, and when computing GPU-busy from the sqlite, union CUPTI_ACTIVITY_KIND_KERNEL with CUPTI_ACTIVITY_KIND_GRAPH_TRACE (+ MEMCPY). Detect graphs via the cudaGraphLaunch vs cudaLaunchKernel API counts.

Gotcha 2 - --capture-range=cudaProfilerApi needs a trigger. It only starts collecting when the app calls cudaProfilerStart. vLLM does that only when launched with --profiler-config.profiler=cuda and you POST /start_profile. Without that, either set --profiler-config.profiler=cuda (preferred - precise window, no restart per capture) or drop --capture-range and use a timed window (--delay/--duration) - but a fixed window is brittle (it can miss steady-state, and a long window can OOM at gpu-mem-util >= 0.85, truncating the report). See inference-decode-step-budget for the endpoint-triggered recipe.

Gotcha 3 - --attach-pid caveats. Attaching to a running PID for CUDA tracing works only when the process has no conflicting CUDA injection (production zymtrace-instrumented pods set CUDA_INJECTION64_PATH and block it - see the ncu skill's "Blocker 1". Latency canaries usually have none). For the decode tier prefer the profiler-endpoint path over attach.

Capture-quality gate (decode captures). Before trusting a decode trace, verify it (a) contains CUDA-kernel data, (b) shows a repeating decode-step pattern (a dominant inter-step idle-gap bucket), and (c) is not prefill- contaminated - drive a CLEAN single-stream workload (tiny prompt + long gen + ignore_eos), not vllm bench serve --random-input-len N.

Gate 0 (precondition) - CUPTI must even initialize (CUDA image-vs-driver skew). On GB300, a CUDA 12.9 serving image against the node's CUDA 13.0 driver makes CUPTI fail to init (CUPTI_ERROR_INVALID_DEVICE), so EVERY CUPTI client (nsys, ncu, vLLM --profiler-config.profiler=torch) records 0 kernels REGARDLESS of the four gates below. This is NOT capture hygiene and the gates won't fix it - it is a CUDA major-version toolkit-vs-driver skew (NOT permission: RmProfilingAdminOnly:0. NOT missing-lib: libcupti.so.12 present + linked. NOT wrong-process). If you get 0 kernels on GB300, grep the kineto/nsys log for CUDA versions. CUPTI/Runtime/Driver FIRST: a 12.x-toolkit / 13.x-driver split needs a CUDA-13-aligned image or zymtrace (non-CUPTI), not more capture tuning. GB300 path: use an operator-owned launch wrapper that copies a Blackwell-capable Nsight Systems 2026.x build from an approved CUDA 13 image into the serving container. This repository does not ship deployment manifests for that wrapper. Record the source image digest, copied binary version, and launch change in the evidence bundle. Do not rely on an older in-image or apt-installed Nsight Systems build for this CUDA 13 path.

An EMPTY cuda_gpu_kern_sum is a CAPTURE-HYGIENE bug, NOT a "cudagraph blind spot" - DO NOT conclude the stack is unprofilable until you validate four things (only after Gate 0 passes). A single empty rep has been wrongly declared a "CUDA-graph blind spot" before. A re-capture on the same production cudagraph stack with fixed capture hygiene returned tens of millions of kernel rows:

1. Flag - --cuda-graph-trace=node is in the nsys argv (Gotcha 1 above), without it graph-resident kernels are opaque GRAPH_TRACE and kern_sum is empty at c>=64.
2. Traffic - a bench is DRIVING load during the capture window. At the throughput tier (c>=64) drive continuous --max-concurrency >=64 traffic during [delay, delay+duration]. An idle/untrafficked window yields an empty rep. (At the decode/c=1 tier this is the CLEAN single-stream workload above.)
3. Rep-size - a real c>=64 rep is hundreds of MB to GB (421 MB @ c=128, 1.0 GB @ c=192). .nsys-rep << ~10 MB => idle/failed capture, RETRY. Never run stats on a tiny rep.
4. Sqlite probe - nsys export --type sqlite then SELECT count(*) FROM CUPTI_ACTIVITY_KIND_KERNEL. A kern_sum SKIPPED message ALONE is not proof. It fires on a near-empty rep. Only after all four hold and it is STILL empty may you escalate to a genuine tooling limit. Mechanical gate: scripts/nsys-validate-capture.sh (checks the flag, rep-size, and KERNEL row count. PASS/RETRY).

Two launch-wrap gotchas that also hit nsys (canon: the ncu capture-hygiene section of inference-kernel-ncu-profile + docs/METHODOLOGY.md): (1) -- env VAR=v python3 makes the profiler target the env process not its python child - set the env on the profiler process itself (VAR=v nsys … -- python3 …), (2) profiler-start-based scoping is unreliable when the harness imports a vLLM direct_register_custom_op module - prefer launch-index/delay scoping.

4. Extract artifacts

BUNDLE=experiments/artifacts/inference-perf-bench/<bundle>/
mkdir -p $BUNDLE/profiles
kubectl -n $NS cp $TARGET_POD:/profiling/capture.nsys-rep \
  $BUNDLE/profiles/capture.nsys-rep -c $SIDECAR
kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- bash -c "
  nsys stats --report cuda_api_sum,gpu_kern_sum,osrt_sum \
    /profiling/capture.nsys-rep --format=csv \
    --output=/profiling/summary
"
kubectl -n $NS cp $TARGET_POD:/profiling/summary_gpu_kern_sum.csv \
  $BUNDLE/profiles/gpu_kern_sum.csv -c $SIDECAR

5. Populate kernel_profile field in inference_perfbench_v1.json

import json, pathlib
bundle = pathlib.Path("$BUNDLE")
ipb = json.loads((bundle / "inference_perfbench_v1.json").read_text())
ipb["kernel_profile"] = {
    "captured_at": "<UTC>",
    "nsys_rep_path": "profiles/capture.nsys-rep",
    "summary_csv_path": "profiles/gpu_kern_sum.csv",
    "duration_s": 120,
    "vllm_pid": $VLLM_PID,
    "sidecar_image": "ghcr.io/cfregly/nsys-sidecar:0.1.0@sha256:3146de96f6022a8cc36f86d1b8c0281cb940e51e2c3dc49c315646ad66ede43d",
    "method": "kubectl-debug-share-processes",
    # populate top_kernels from the CSV's "name" + "samples" columns
}
(bundle / "inference_perfbench_v1.json").write_text(json.dumps(ipb, indent=2))

The inference_perfbench_v1 schema treats kernel_profile as an optional dict - bundles without it are unaffected.

6. Cleanup

kubectl -n $NS exec $TARGET_POD -c $SIDECAR -- rm -rf /profiling/*
# The sidecar container will terminate after the 3600s sleep; pod
# auto-cleans the ephemeral container.

Output schema

The capture produces three files under <bundle>/profiles/:

The CSV columns from nsys stats --report gpu_kern_sum:

Time (%), Total Time (ns), Instances, Avg (ns), Med (ns), Min (ns), Max (ns), StdDev (ns), Name

The renderer + analyze-zymtrace-workload skill (Phase 4d) join this CSV with the zymtrace per-category breakdown via the Name column's canonical-substring matching.

Cross-skill join

The analyze-zymtrace-workload skill (Phase 4d) reads the kernel_profile field when present and joins:

• zymtrace categorical share: e.g. "FMHA = 14.2%"
• nsys per-kernel share: e.g. "fmha_v2_kernel_: 11.8%, fmha_v2_kernel_paged: 2.4%"

Showing the operator both numbers at once closes the loop on "is the hot category one kernel or three?" - answerable from zymtrace alone is "unknown beyond the category".

Cost + risk

• ~5% per-pod overhead during the 120s capture window
• nsys-rep files are typically 50-200 MB (manageable)
• Sidecar attach is non-disruptive (no vllm restart)

Pair this skill with evidence-bundle-init when starting a new investigation so the profiles/ subdir lands in the right place from the start.

Pairs with

• inference-decode-step-budget - FAST, restart-free c=1/low-concurrency decode-step budget via vLLM's /start_profile endpoint. Use it (not this skill) when the question is "where does decode time go / is it kernel- or host-bound".
• zymtrace-anchored-query - kernel-name resolution + sample-share from ClickHouse, no cluster mutation. Run FIRST. An empty result right after a bench is usually ClickHouse ingest lag (async flush), not absence - wait + requery for the freshest data (see server/docs/zymtrace-query-hygiene.md).
• inference-kernel-ncu-profile - per-kernel hardware counters (occupancy, regs, smem, DRAM BW, roofline) using the SAME sidecar image, scoped via --kernel-name to avoid the 10-100x kernel-replay slowdown. Run AFTER nsys to interpret hot-kernel ROOT CAUSE (memory-bound vs compute-bound vs occupancy-limited).

Full-context reporting (no bare numbers)

Per the canon "Every performance number carries its full context (no bare numbers)" (docs/METHODOLOGY.md "Full-context reporting"): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare tok/s / TPOT / BW / %SoL / speedup is a defect - it cannot set a default, ship a config, or appear in a report.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP, parallel_strategy.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, bench backend, serving engine.
• Grounding: %SoL (+ ceiling key from configs/sol-ceilings.yaml - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named.
• Per-number exact shape (no smoothing): when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. c=1 @ ISL1024/OSL256 + c=64 @ ISL4096/OSL512, NOT one shared "random").

Per docs/METHODOLOGY.md "Speed-of-light framing", kernel-time hot-spot interpretation MUST cite the natural ceiling each kernel family is bound by, sourced from configs/sol-ceilings.yaml:

• NCCL kernels → nvlink5_tbps (1.8 TB/s bi-dir per GPU on B200)
• MoE / BMM-NVFP4 → nvfp4_dense_pflops (9 PFLOPS dense on B200)
• FMHA → hbm3e_tbps (8 TB/s on B200)
• cuBLAS / Triton-fused → bf16_dense_pflops (2.25 PFLOPS on B200)

nsys gpu_kern_sum gives wall-clock time per kernel. The %SoL conversion needs FLOPS / bytes per kernel which ncu captures, not nsys. When a nsys finding flags a hot kernel that needs proper %SoL interpretation, the next step is inference-kernel-ncu-profile on that kernel name - see the "Pairs with" entry above.

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured result, its output MUST end by naming the next perf lever, its expected unlock (direction + rough magnitude), and the gate that proves/refutes it, per docs/METHODOLOGY.md "Always be grinding". A measured win is the new floor, not the finish -- so do everything we can to find the next BREAKTHROUGH: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, perftunereport value_view), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's HANDOFF.md. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.

Kernel rubric (K/R/H/P/A)

When this nsys capture backs a custom-kernel comparison, apply the kernel rubric (docs/METHODOLOGY.md "Kernel-work classification"). Record (K,R,H,P,A) for the candidate AND the named baseline in the bundle's SOURCE.md/summary.md. nsys gives you the K and R axes directly (which op, library vs Triton vs CUDA-graph-captured) and the absolute per-kernel duration that feeds the P-axis comparison - but it cannot prove the H axis (tensor-core engagement) or a true %SoL. Defer the H + P proof to inference-kernel-ncu-profile, which is the gate's enforcement point. Reminder: a win over a strictly-lower-H/R baseline (e.g. beating generic Triton when production runs the sm100f tensor-core library) is a DRAFT, never a VERDICT - it fails the "Fair baseline" clause. When the campaign reaches L4 (an ncu roofline renders), the structured krhpa: block in config.yaml is required by publish_to_lake (see the YAML example in inference-kernel-ncu-profile). Prose in SOURCE.md/summary.md alone no longer satisfies the gate.