Inference Perf Baseline Bridge

inference-perf-baseline-bridge

Purpose

Translate the output of an inference-perf-bench (or its colloquial alias ai-bench) run into the shape the workload-agnostic perf-baseline-record / perf-baseline-diff registry expects. The bridge fixes three pain points:

1. The upstream perf-bench skill's report is a markdown table per model - not a JSON payload the registry can hash and diff.
2. The metric set (TTFT p50/p95/p99, ITL p50, tok/s, tok/s/user, request latency p50, prefix cache hit rate delta, GPU cache utilization peak) needs a stable schema name so future diffs find apples-to-apples baselines.
3. Per-metric regression tolerances differ - TTFT regressions of 5% are usually real, throughput regressions of 3% may be capacity contention, prefix cache hit rate movements should be expressed as absolute percentage points (not percent-of-percent).

This bridge does NOT add a new MCP verb. It wraps the existing mcp__profile_and_optimize__perf_baseline_record and mcp__profile_and_optimize__perf_baseline_diff verbs with a fixed schema kind (inference_perfbench_v1) and operator-tunable per-metric tolerances.

Schema: inference_perfbench_v1

{
  "schema": "inference_perfbench_v1",
  "model": "<served-model-name>",
  "model_path": "<HF-org/repo>",
  "deployment": {
    "namespace": "<k8s-namespace>",
    "cluster": "<kube-context>",
    "service": "<svc-name>",
    "vllm_version": "<v0.NN.M>",
    "tensor_parallel_size": <int>,
    "quantization": "<NVFP4 | FP8 | BF16 | FP4 | INT4>",
    "kv_cache_dtype": "<fp8_e4m3 | bf16 | ...>",
    "speculative_decoding": "<eagle3 | mtp | none>"
  },
  "workload": {
    "dataset": "<hf-org>/replay-playback",
    "split": "<2025_07 | 2025_04 | 2025_01 | all>",
    "conversation_num": <int>,
    "concurrency_sweep": [<int>, ...],
    "input_token_band": "<1k | 10k | 100k | mixed>",
    "output_token_band": "<1k | 1.5k | 2k | mixed>"
  },
  "metrics": {
    "<concurrency>": {
      "throughput_toks_per_s": <float>,
      "ttft_p50_ms": <float>,
      "ttft_p95_ms": <float>,
      "ttft_p99_ms": <float>,
      "itl_p50_ms": <float>,
      "tps_per_user": <float>,
      "request_latency_p50_ms": <float>,
      "errors": <int>
    }
  },
  "server_metrics_delta": {
    "prefix_cache_hit_rate_pre": <float>,
    "prefix_cache_hit_rate_post": <float>,
    "gpu_cache_usage_perc_peak": <float>,
    "num_requests_running_peak": <int>,
    "avg_generation_throughput_toks_per_s": <float>
  },
  "kernel_class_gpu_pct": {
    "fp8_bmm": <float|null>,
    "tp_all_reduce": <float|null>,
    "cuda_event_sync": <float|null>,
    "mla_attention": <float|null>,
    "moe_routing_finalize": <float|null>,
    "per_token_group_quant": <float|null>,
    "<other-class>": <float|null>
  },
  "cpu_spinpoll_pct": <float|null>
}

Optional fields

The two top-level fields kernel_class_gpu_pct and cpu_spinpoll_pct are optional. They surface profile-grade signals that the upstream perf-bench runbook does not produce on its own - they are populated when the operator also captures a zymtrace or Nsight-Systems profile against the same time window.

Field shapes and origins:

• kernel_class_gpu_pct is an open map of kernel-class name to percent of GPU time. The canonical class set is the one typically observed in a plain-FP8 MoE serving profile: fp8_bmm, tp_all_reduce, cuda_event_sync, mla_attention, moe_routing_finalize, per_token_group_quant. Classes outside this set go under arbitrary <other-class> keys (free-form). The map is derived from a GPU-side event_kind = 'cuda' zymtrace query grouped by kernel-name regex. The source bucket-rules live in the zymtrace-anchored-query skill's "Kernel-class bucketing" appendix.
• cpu_spinpoll_pct is a single scalar: the percent of CPU on-cpu samples that fall inside vLLM's spin-poll IPC path (shm_broadcast.py::{acquire_read,wait,memory_fence,timeout_ms,should_warn,dequeue}
  • utils.py::sched_yield). Populated from the same zymtrace bundle's CPU-side event_kind = 'on_cpu' query.

Diff semantics on null: when either side of a per-metric comparison is null, the bridge skips that metric in Phase 2: per-metric diff with tolerances and emits a NULL_LEGACY_BASELINE row in the diff output so the operator can see which fields were unmeasured. This keeps existing inference_perfbench_v1 baselines registered before these fields existed diff-compatible with new bundles that include them - the registry never breaks.

When a true breaking change is needed (e.g. metric-shape restructure of metrics.<concurrency>), the schema name will bump to inference_perfbench_v2. Additive-only changes stay on _v1.

Default per-metric tolerances

Operator can override any of these via --tolerance.<metric> <value> on the command line. The two optional fields' tolerances are only evaluated when both sides of the diff have non-null values. Otherwise the metric is reported as NULL_LEGACY_BASELINE and skipped (see "Optional fields" above).

When to use

• After an inference-perf-bench run completes (Phase 9 of the upstream runbook).
• Periodic regression sweep - weekly cron diffs each registered baseline against the most recent run.
• Pre-promotion gate - block staging or prod promotion if the diff verdict is RED.
• Pairing perf vs quality - register the baseline with --notes "GPQA=<score>. MMLU-Pro=<score>" from the inference-model-eval run so future diffs can confirm the regression is not masked by a quality gain (or vice versa).

Do not use this skill for:

• MLPerf training step-time / MFU baselines - use perf-baseline-record directly with --family <bench> and a custom schema.
• One-off diagnostic measurements - register a baseline only when the measurement is worth remembering.

Example prompts

• "Register the kimi-k25 perf-bench result as the new baseline."
• "Diff this morning's deepseek-v4 ai-bench against last week's baseline."
• "Inference perf regression check on glm-5-fp8 with tolerance.tps=5%."
• /inference-perf-baseline-bridge record --model kimi-k25 --source experiments/artifacts/inference-perf-bench/<run-id>/
• /inference-perf-baseline-bridge diff --baseline experiments/artifacts/perf-baselines/inference/kimi-k25/2026-05-19T20:00:00Z/ --current experiments/artifacts/inference-perf-bench/<run-id>/

Prerequisites

1. Source bundle path - --source <path> pointing at an experiments/artifacts/inference-perf-bench/<run-id>/ directory.
2. Model name - --model <served-model-name> (matches the value returned by the model's /v1/models endpoint).
3. PROFILE_AND_OPTIMIZE_REPO_ROOT for the registry path.

Interaction style

Mostly autonomous. One pause: confirm the per-metric tolerances and the registry path before write (record) or before computing the diff verdict (diff).

Workflow

Mode 1: record

Phase 1: parse the perf-bench bundle into the schema shape

Read the upstream perf-bench output:

• experiments/artifacts/inference-perf-bench/<run-id>/perf-bench-report-*.md - parse the per-concurrency rows.
• experiments/artifacts/inference-perf-bench/<run-id>/aiperf-c<N>.log - extract the headline metrics per concurrency.
• experiments/artifacts/inference-perf-bench/<run-id>/<model>-metrics-pre.prom and <model>-metrics-post.prom - extract vllm:prefix_cache_hit_rate, vllm:gpu_cache_usage_perc, vllm:num_requests_running, vllm:avg_generation_throughput_toks_per_s.

If the operator also captured a paired profile bundle (zymtrace or Nsight) against the same window, source the two optional fields:

• experiments/artifacts/zymtrace-bundles/<run-id>/response.json (or any sibling bundle that follows the zymtrace-anchored-query layout) - derive kernel_class_gpu_pct from the GPU-side event_kind = 'cuda' query response, bucketed per the kernel-class regex appendix in that skill. Derive cpu_spinpoll_pct from the CPU-side event_kind = 'on_cpu' response by summing samples whose py_file matches shm_broadcast.py or utils.py::sched_yield.
• If no paired profile bundle exists, emit null for both optional fields. The diff verb will skip them per the "Optional fields" rule above. The registry record stays valid.

Emit inference_perfbench_v1.json to a scratch path, sha256sum it.

Phase 2: register

mcp__profile_and_optimize__perf_baseline_record with:
  args: ["--family", "inference",
         "--measurement", "<model>-perfbench-v1",
         "--source", "<scratch-path-to-inference_perfbench_v1.json>",
         "--direction", "two-sided",
         "--unit", "structured-json",
         "--notes", "<operator-supplied notes; e.g. GPQA + MMLU-Pro scores from inference-model-eval>",
         "--json"]

The verb writes the registry entry under experiments/artifacts/perf-baselines/inference/<model>-perfbench-v1/<UTC-ts>/ (per the registry layout the upstream verb already implements).

Phase 3: report

Print:

• Registry path.
• Source SHA-256.
• Headline metrics at concurrency 16 (the canonical operating point for routine comparison).
• Cross-link to inference-perf-baseline-bridge diff for future regression checks.

Mode 2: diff

Phase 1: resolve baseline + current

• Read <baseline>/baseline.json - confirm family == "inference" and measurement == "<model>-perfbench-v1".
• Read <baseline>/source-snapshot.<ext> - load the baseline's inference_perfbench_v1.json.
• Parse the current bundle into the same schema shape (Phase 1 of record mode).

Phase 2: per-metric diff with tolerances

For every concurrency cell in metrics.<concurrency>, compute:

• delta = current - baseline
• delta_pct = 100 * delta / baseline (for percent metrics)
• delta_abs = current - baseline (for absolute metrics: cache hit rate)
• Per-metric verdict: GREEN if |delta| is within the tolerance for that metric (regression direction respected), RED otherwise.

For server_metrics_delta, apply the tolerances at the absolute level: a prefix_cache_hit_rate_post drop of more than 2 percentage points is RED. A peak gpu_cache_usage_perc rise of more than 5 percentage points is RED.

Phase 3: overall verdict

• GREEN: every metric at every concurrency is within tolerance.
• YELLOW: 1-2 metrics exceed tolerance OR exactly one concurrency level shows multiple regressions.
• RED: 3+ metrics exceed tolerance OR the headline metric (throughput_toks_per_s at concurrency 16) regresses by more than its tolerance.

Phase 4: write the diff bundle

Delegate to mcp__profile_and_optimize__perf_baseline_diff (the existing verb already writes the diff under experiments/artifacts/perf-baseline-diffs/inference/<model>-perfbench-v1/<UTC-ts>/):

mcp__profile_and_optimize__perf_baseline_diff with:
  args: ["--baseline", "<baseline-registry-entry-dir>",
         "--current", "<scratch-path-to-current-inference_perfbench_v1.json>",
         "--tolerance-percent", "5",
         "--json"]

The wrapper supplies 5 as the umbrella tolerance for the underlying verb's structured-JSON shape. The per-metric verdict above is the authoritative classification this skill returns.

Phase 5: recommend

• GREEN: "no inference perf regression detected. Safe to promote."
• YELLOW: "spot regression on N metrics. Recommend re-running the perf-bench against the same dev cluster cohort to rule out noise, then re-diff."
• RED:
  • If headline throughput_toks_per_s regressed: prompt the operator to check for cohort contention via prometheus-anchored-query on vllm:num_requests_running for the same time window.
  • If TTFT regressed but throughput didn't: suggest checking scheduler / queue depth (vllm:num_requests_running peak + vllm:gpu_cache_usage_perc peak deltas).
  • If prefix cache hit rate regressed: suggest the --connection-reuse-strategy sticky-user-sessions flag - a hit-rate drop is often a routing problem (cross-region hops breaking session affinity), not a model-side issue.

Safety

• Read-only on the registry. Diff mode never writes to experiments/artifacts/perf-baselines/. It writes only to experiments/artifacts/perf-baseline-diffs/.
• No new MCP verbs. This skill wraps existing mcp__profile_and_optimize__perf_baseline_record and mcp__profile_and_optimize__perf_baseline_diff verbs. The bundled MCP surface tool count stays unchanged.
• Schema mismatch fails fast. If the baseline's schema field is not inference_perfbench_v1, diff mode stops and asks. No cross-schema fuzzy diffing.
• Provenance preserved. Every record / diff carries the perf-bench bundle's source SHA-256 + operator + cluster + profile_and_optimize SHA, per the server/AGENTS.md reproducibility-grade-evidence rule.

Full-context reporting (no bare numbers)

Per the methodology canon "Every performance number carries its full context (no bare numbers)" (docs/METHODOLOGY.md): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare tok/s / TPOT / BW / %SoL / speedup is a defect - it cannot set a default, ship a config, or appear in a report.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP, parallel_strategy.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, delivery (image|overlay|patchedVllm|infr-patch), bench backend, serving engine. A number is evidence only for its own delivery -- never cite cross-tier.
• Grounding: %SoL (+ ceiling key from configs/sol-ceilings.yaml - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named.
• Per-number exact shape (no smoothing): when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. c=1 @ ISL1024/OSL256 + c=64 @ ISL4096/OSL512, NOT one shared "random").

See server/AGENTS.md "Speed-of-light framing". When this bridge records / diffs a perf-bench run, the baseline record SHOULD carry a sol_pct field per the perf-baseline-record schema, and the diff SHOULD report SoL delta alongside the absolute throughput / latency delta. Peaks are sourced from configs/sol-ceilings.yaml - never inlined.

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured result, its output MUST end by naming the next perf lever, its expected unlock (direction + rough magnitude), and the gate that proves/refutes it, per "The Grind Mandate" (server/AGENTS.md + docs/METHODOLOGY.md). A measured win is the new floor, not the finish -- so do everything we can to find the next BREAKTHROUGH: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, perftunereport value_view), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's HANDOFF.md. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.

Source-of-truth references

• The pair this bridge wraps: perf-baseline-record, perf-baseline-diff.
• Perf-bench input: inference-perf-bench (also reachable via the ai-bench colloquial alias).
• Quality counterpart: inference-model-eval.
• server/AGENTS.md - fail-fast + provenance rules.