# Run Aiperf Benchmark

> Launches, monitors, debugs, and collects one run-scoped AIPerf Kubernetes benchmark without changing its workload semantics. Use after perf.yaml and aiperf-config.yaml have been configured for a smoke-tested Dynamo deployment.

- Skill: `ai-dynamo-dynamo/run-aiperf-benchmark` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ai-dynamo-dynamo/run-aiperf-benchmark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ai-dynamo-dynamo/run-aiperf-benchmark/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: Apache-2.0
- Author: ai-dynamo (https://skillmd.com/u/ai-dynamo-dynamo)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/ai-dynamo-dynamo/run-aiperf-benchmark

---


# Run AIPerf Benchmark

<!--
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

Execute the configured Job and preserve operational evidence. Do not interpret performance.

Read `agent-docs/rules/execution/deployment.md`, `agent-docs/rules/execution/logging.md`,
`agent-docs/rules/execution/run-artifacts.md`, `agent-docs/rules/benchmarking/benchmark-isolation.md`,
`agent-docs/rules/benchmarking/comparison-uncertainty.md`,
`agent-docs/rules/benchmarking/evidence-eligibility.md`, `agent-docs/rules/benchmarking/series-boundaries.md`, and the
exact active benchmark plan first.

## Preflight

- Confirm the selected Kubernetes context and namespace.
- Verify the plan path, SHA256, series ID, and performance question against the run-scoped configuration.
- Server-dry-run `perf.yaml` and verify its Job name from `metadata.name`.
- Verify the frontend service is reachable in cluster and `/v1/models` exposes the configured served model.
- Verify the AIPerf config, workload trace, tokenizer, PVC/mounts, image, and referenced secrets are available to the
  benchmark pod.
- Confirm the AIPerf measurement is configured to finish in 30 minutes or less and uses one repetition by default.
- For an additional valid-run repetition, require the prior analysis to record why the repeat is necessary and worth
  its GPU cost. Do not repeat only to obtain confidence intervals. The once-per-series noise-floor pilot (n=3)
  required by `comparison-uncertainty.md` is pre-authorized and arrives through the normal `repeat_decision:
  necessary` path with a rationale naming the series pilot; it needs no further justification.
- Confirm no other benchmark is targeting the same candidate.
- Every kubectl invocation this skill drives pins `--context "${KUBE_CONTEXT}"` and `-n "${NAMESPACE}"`,
  bounds every wait with an explicit timeout, distinguishes job `Failed` from timeout (poll the job's true
  condition rather than waiting on `Complete` alone), and chains commands so a failure never flows into the next
  step — the same discipline `deploy-dynamo-recipe`'s scripted blocks encode.
- Record neighbour occupancy in `benchmark_execution.json` at run start and end (the pods scheduled on the serving
  nodes, e.g. `kubectl --context "${KUBE_CONTEXT}" get pods -A -o wide` filtered to those nodes), so
  `comparison-uncertainty.md`'s like-for-like and transition checks have a recorded condition to read.

## Execute And Monitor

1. Apply the run-scoped Job.
2. Wait with a bounded timeout and coarse polling.
3. Monitor Job conditions, pod phase/restarts, recent events, and the relevant AIPerf log tail.
4. On completion, copy the complete AIPerf output directory unchanged into
   `<DEPLOY_ROOT>/benchmark/raw_aiperf/` before deleting the Job or pod.
5. Record exact commands, Job/pod names, start/end timestamps, configured and actual measurement duration, whether this
   was the default run or an approved repeat, repeat rationale, AIPerf source/runtime versions, exit status, retry
   history, and artifact paths in `benchmark_execution.json`.

## Debug By Ownership

| Failure | Action |
|---|---|
| Endpoint, model, or deployed workload failure | Record a handoff to `recipe-deployer`; do not patch the DGD here |
| Invalid trace or AIPerf settings | Return to `configure-aiperf-benchmark` |
| Repairable invalid evidence | Consume the audit blockers from `analyze-aiperf-results`, rerun the active series unchanged without overwriting prior raw artifacts, record the retry, then return to `analyze-aiperf-results` |
| Job scheduling, mount, image, or artifact-copy issue | Repair only the run-scoped benchmark manifest and record why |
| AIPerf client/tool failure | Use the pinned AIPerf docs/source, repair without changing benchmark semantics |
| Repeated identical failure | Stop and record the blocker |

Do not reduce request count, remove difficult trace rows, relax SLOs, lower load, switch schedule mode, or alter
ISL/OSL to make a failed benchmark finish. Such a change creates a different benchmark plan and comparison series.
Do not launch another valid run because a small delta was classified as noise or because confidence intervals are
absent.

## Output Status

Set `benchmark_execution.json.status` to `completed`, `failed`, or `blocked`. A completed Job is not automatically a
valid benchmark; validity belongs to the audit phase of `analyze-aiperf-results`. Record the exact benchmark-plan path,
SHA256, series ID, and performance question in the execution record.

Keep logs only when they explain a failure beyond the concise execution ledger. Do not retain routine successful pod
logs or broad cluster snapshots.