Inference Perf Synthesize

inference-perf-synthesize

Purpose

The four profiler skills each answer a different slice of "where does the time go and why":

Do NOT conclude "kernel at-roof / no lever" from SM-busy% (ncu --set basic or DCGM SM-active). An 88-92% SM-busy can hide a kernel at <15% of its FLOP-roofline (persistent/split-K spin). Decide at-roof vs headroom only from ncu --set full achieved-TFLOPS vs the FLOP ceiling. Worked example: a DeepSeek-V4-Flash FP8 GEMM read 88-92% SM but was 1-14.5% of FP8 FLOP-SoL -> the synthesized recommendation was "throughput tier", not "kernel rewrite". (Canon: the ncu capture-hygiene section of inference-kernel-ncu-profile.)

Run in isolation they produce four disconnected artifacts. This skill fuses them into one ranked recommendation ledger: it reconciles the four views (e.g. zymtrace says category X is hot -> DCGM confirms X is byte-bound -> ncu shows X reaches only Y% of the tensor-core roofline -> nsys shows a host gap before it), turns that into a ranked list of addressable levers (config knobs AND concrete vLLM source changes), and stages each lever as an A/B experiment so the recommendation is proven on this exact workload before it is believed.

It does NOT replace the four skills -- it consumes their outputs. Run the profilers first (or let inference-model-optimize Phase 2 drive them), then run this to synthesize.

When to use

• After profiling a deployment with 2+ of the four profilers, to get a single prioritized "what to change" answer instead of four separate reports.
• inference-model-optimize Phase 2.5 (diagnose the baseline -> recommend levers that seed the tuning phase) and Phase 8.5 (post-champion: what's left on the table + future-work levers).
• When a stakeholder asks "why is it slow and what do we change" and the answer must be data-backed and explainable, not asserted.

Do not use this skill for:

• Capturing a profile -- that is the four profiler skills. This skill reads what they captured.
• A which-config sweep -- that is inference-tune-sweep (this skill feeds it the candidate levers).
• A single-profiler reading where no cross-view is needed -- just use that profiler's skill.

Example prompts

• "Synthesize the nsys + ncu + zymtrace + DCGM captures into ranked recommendations."
• "What should I change in vLLM to make this faster, backed by the profiles?"
• "Holistic perf analysis of the gemma4 baseline -- explain the bottleneck with data."
• /inference-perf-synthesize --bundle <evidence-bundle> --focus latency

Prerequisites

The skill fails closed if any of these are not satisfied.

1. An evidence bundle with at least two of the four profiler artifacts present (*.nsys-rep / *.ncu-rep / zymtrace TSVs / dcgm_correlation.json). It will not synthesize from a single source (that is not a cross-view).
2. The capture-validation gates passed for each source (an empty nsys rep is a capture bug, not a finding -- see the nsys capture-validation gate). The skill refuses to draw a conclusion from an unvalidated/empty artifact.
3. For any recommendation it stages as an A/B: a writable experiment-prefixed overlay path + an idle GPU node (plain-K8s GB300 / legacy Slurm-on-K8s) (the A/B itself runs via inference-tune-sweep / the controlled-A/B harness).

Capture orchestration (how to actually get the 4 artifacts)

Getting the four artifacts is itself constrained -- these rules are hard-won. Encode them so a capture is right the first time:

• One pod cannot run all four profilers. ncu and the zymtrace CUDA injection (CUDA_INJECTION64_PATH) are mutually exclusive, nsys needs the sidecar image or vLLM /start_profile. So a full capture is 2-3 coordinated pods: (A) the bench pod with the zymtrace injection ON + vLLM torch profiler (VLLM_TORCH_PROFILER_DIR + /start_profile//stop_profile window), (B) a SEPARATE nsys launch-wrap pod (injection cleared, --cuda-graph-trace=node), (C) a SEPARATE ncu sister pod (--set basic, --kernel-name scoped). DCGM is correlated POST-HOC over the bench window (no in-pod agent).
• Use a submit-and-queue self-driving Job on a contended cluster (per server/AGENTS.md): the capture pod embeds serve + bench + window-stamp, writes to the RWX PVC, and EXITS to free the GPU, backoffLimit re-queues through preemption, guard idempotency on the result file (not a marker that could be touched on failure). Make the bench variance-controlled (>=3 trials/concurrency, same pod => same node) so the baseline is VERDICT-grade, not a single-shot DRAFT.
• Offline-tokenizer gotcha (L1): in an HF_HUB_OFFLINE=1 pod, vllm bench serve --model <hf-id> will try to fetch the tokenizer from huggingface.co and fail with LocalEntryNotFoundError. ALWAYS pass --tokenizer <local-weights-dir> so the bench client uses the on-disk tokenizer. (This silently produced zero results until caught.)

Workflow

Phase 1: inventory + validate the captured artifacts

Read the bundle. List which of the four profilers produced a valid artifact (apply each capture-validation gate). Record the sol_rigor available per source. Refuse to proceed if fewer than two valid sources exist.

Phase 2: reconcile the four views

Build a per-kernel-family / per-phase table that joins the sources:

• zymtrace sample-share (L1) -> which category/kernel is hot, CPU vs GPU side.
• DCGM byte/FLOP (L3) -> is that category byte-bound or compute-bound at the workload level.
• ncu (L4) -> for the dominant kernel: achieved roofline %, tensor-core engaged?, occupancy / stall reason / register or smem pressure.
• nsys -> absolute duration + whether a host gap precedes it (host-bound vs GPU-bound), cudagraph-captured or not.

Flag contradictions (e.g. zymtrace says hot but ncu says near-roofline -> not addressable) -- a contradiction is itself a finding, not something to smooth over.

Phase 3: emit the ranked recommendation ledger

For each addressable lever, write a ledger row:

rank | lever | type(config|vllm-src|kernel) | expected_effect | evidence_artifacts[] | source_refs[] | sol_rigor | confidence(DRAFT)

source_refs[] is MANDATORY for any vllm-src / kernel lever: the exact vLLM/SGLang branch + commit (resolved to a GitHub URL) and delivery (image|overlay|patchedVllm|infr-patch) from the bundle's experiment_provenance_v1 block + the campaign's source registry, so a reviewer can open the patch that produced the win. A source-code recommendation with no source_refs[] is not actionable. Code-under-test provenance match: the expected_effect number MUST come from an evidence_artifacts[] campaign whose provenance delivery+commit matches the recommendation's source_refs[] -- an overlay/offline-prepped campaign cited as the benefit of an infr-patch recommendation is a cross-tier DRAFT defect (even when the kernels match). Cite the patch's own run.

• type=config: a vllm.extraArgs / cudagraph / kv-cache-dtype change -> hand to inference-tune-sweep.
• type=vllm-src: a concrete vLLM code change, staged as a runtime overlay per the delivery ladder (server/AGENTS.md "Experiment delivery ladder"): a subPath ConfigMap (overlay_mode: subpath) for a few files, or the initContainer patch-set (overlay_mode: patchset-initcontainer) for a whole patch set on a pullable base -- include the file + the diff sketch + which artifact motivates it.
• type=kernel: a custom-kernel opportunity -- classify it with the K/R/H/P/A rubric and name the production-representative baseline (a win over a strictly-lower-H/R baseline is a DRAFT, never a VERDICT).

Every row MUST list evidence_artifacts[] (the exact .nsys-rep / .ncu-rep / zymtrace TSV / dcgm_correlation.json / kernels.json paths). A row with no artifact citation is rejected -- this is the explainability contract.

Phase 4: prove each lever (A/B), do not assert

Take the top-ranked levers and stage each as an experiment:

• config -> a campaign_run cell or a controlled A/B.
• vllm-src -> an overlay deploy (experiment-prefixed) + a same-node controlled A/B vs the unpatched arm, both under matching cudagraph_mode.

Gate each with perf-baseline-diff (or perf_baseline_diff). A lever is promoted from DRAFT (predicted) to VERDICT (proven) only after a same-node + >=3-trial + metric-isolated A/B confirms it. A lever whose A/B comes back within noise is recorded as "tried, null" -- kept in the ledger so the next operator does not re-explore it.

Phase 5: persist + publish

• Write findings/synthesis-<ts>.md to the evidence bundle (the ranked ledger + the reconciliation table + the A/B verdicts), via findings_record.
• Feed the ledger into the perftunereport campaign as a recommendations view so it renders in the PDF and lands in the perf-lake under campaign=<run-id> with focus + sol_rigor recorded.
• Record the campaign + s3://perf-lake/... paths back into the bundle's SOURCE.md/summary.md.

Explainability contract (mandatory)

Per server/AGENTS.md "All attribution claims must be matched with collected profile data", every recommendation and every where/why verdict this skill emits carries:

1. the backing artifact path(s),
2. the sol_rigor tier of that evidence (L1 zymtrace-proxy / L3 DCGM / L4 ncu),
3. a DRAFT/VERDICT tier (DRAFT = predicted from profiles. VERDICT = proven by an A/B).

A claim missing any of the three is not emitted. An empty/failed capture is a capture bug to fix, never evidence of "unprofilable" -- clear the capture- validation gate first, then conclude. (For zymtrace specifically, an empty-now right after the bench is usually ClickHouse INGEST LAG, not a bug -- wait + requery for the freshest data. See server/docs/zymtrace-query-hygiene.md.) (One genuine environmental exception seen on GB300: a CUDA 12.x image on a CUDA 13.x driver skews CUPTI -> CUPTI_ERROR_INVALID_DEVICE -> 0 kernels for ALL CUPTI clients regardless of hygiene. Grep CUDA versions. CUPTI/Runtime/Driver -> use a CUDA-13 image or zymtrace.)

Verdict rigor (DRAFT vs VERDICT)

Synthesis output is predictive (DRAFT) by construction -- it ranks levers from profile evidence. A lever becomes a VERDICT only after Phase 4's A/B proves it (same-node, >=3 trials, metric-isolated median TPOT/ITL for latency, against the production-representative baseline). Never report a predicted speedup as achieved.

Kernel rubric (K/R/H/P/A)

Any type=kernel recommendation records the candidate AND the named production baseline's (K,R,H,P,A) coordinates per server/AGENTS.md "Kernel rubric". The H + P proof (tensor-core engagement + roofline %) comes from the ncu artifact, a win over a strictly-lower-H/R baseline is a DRAFT, never a VERDICT.

Safety

• No silent fallbacks / no fabricated evidence -- refuses to synthesize from < 2 valid sources or from an unvalidated capture. Rejects any uncited claim.
• A/B before believe -- a vllm-src recommendation is staged as an experiment-prefixed overlay and proven on this workload. It is never applied to a standing deploy or asserted from first principles.
• Experiment isolation + scheduling -- all A/B arms are experiment-prefixed + experiment=<id-slug> labeled and torn down by label. Scheduled per the cluster profile (plain-K8s default-scheduler / the Slurm scheduler on legacy Slurm-on-K8s).
• No external posting of recommended vLLM changes (no upstream PR) without explicit per-turn operator approval.

Source-of-truth references

• The four profiler skills (inputs): inference-kernel-profile, inference-kernel-ncu-profile, analyze-zymtrace-workload, inference-dcgm-correlate.
• inference-tune-sweep + inference-perf-tune-report -- where the staged A/Bs run and the recommendations get published. The ranked levers here become the cross-engine variant arms. The proven survivors flow to perftunereport champion_select (the baseline-vs-top-X production pick + page 8).
• server/AGENTS.md -- attribution-must-be-profiled, speed-of-light framing, DRAFT-vs-VERDICT, kernel rubric.

Full-context reporting (no bare numbers)

Per the methodology canon "Every performance number carries its full context (no bare numbers)" (docs/METHODOLOGY.md): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare tok/s / TPOT / BW / %SoL / speedup is a defect - it cannot set a default, ship a config, or appear in a report.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP, parallel_strategy.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, bench backend, serving engine.
• Grounding: %SoL (+ ceiling key from configs/sol-ceilings.yaml - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named.
• Per-number exact shape (no smoothing): when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. c=1 @ ISL1024/OSL256 + c=64 @ ISL4096/OSL512, NOT one shared "random").

If this skill produces a measurement (tok/s, latency, %SoL, speedup), follow the rigor discipline: capture L1 zymtrace + L3 DCGM (L4 ncu where feasible) Speed-of-Light and publish --strict. Canonical map: server/AGENTS.md "Rigor discipline index" / docs/METHODOLOGY.md. Skills that do not produce measurements are exempt (server/AGENTS.md "Speed-of-light framing").

Asset validation (review + FAIL LOUD)

Every asset this skill emits (findings doc / recommendation ledger / perfreport view) is held to server/AGENTS.md "Validate every generated asset" (docs/METHODOLOGY.md): the generator FAILS LOUDLY on missing/bad data (a recommendation with no backing profile artifact, an empty/degenerate fused view, unknown/null where a value is required) -> raise / flag loudly, never a silent placeholder or an unbacked recommendation, and the agent REVIEWS the rendered doc/ledger for human-sense + 100% accuracy (every recommendation cites its backing artifact + sol_rigor tier, the A/B staging is sound) and rebuilds it if wrong -- never ships a wrong/confusing synthesis with a caveat.

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured result, its output MUST end by naming the next perf lever, its expected unlock (direction + rough magnitude), and the gate that proves/refutes it, per "The Grind Mandate" (server/AGENTS.md + docs/METHODOLOGY.md). A measured win is the new floor, not the finish -- so do everything we can to find the next BREAKTHROUGH: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, perftunereport value_view), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's HANDOFF.md. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.