Run Aiperf Benchmark

Run AIPerf Benchmark

Execute the configured Job and preserve operational evidence. Do not interpret performance.

Read agent-docs/rules/execution/deployment.md, agent-docs/rules/execution/logging.md, agent-docs/rules/execution/run-artifacts.md, agent-docs/rules/benchmarking/benchmark-isolation.md, agent-docs/rules/benchmarking/comparison-uncertainty.md, agent-docs/rules/benchmarking/evidence-eligibility.md, agent-docs/rules/benchmarking/series-boundaries.md, and the exact active benchmark plan first.

Preflight

• Confirm the selected Kubernetes context and namespace.
• Verify the plan path, SHA256, series ID, and performance question against the run-scoped configuration.
• Server-dry-run perf.yaml and verify its Job name from metadata.name.
• Verify the frontend service is reachable in cluster and /v1/models exposes the configured served model.
• Verify the AIPerf config, workload trace, tokenizer, PVC/mounts, image, and referenced secrets are available to the benchmark pod.
• Confirm the AIPerf measurement is configured to finish in 30 minutes or less and uses one repetition by default.
• For an additional valid-run repetition, require the prior analysis to record why the repeat is necessary and worth its GPU cost. Do not repeat only to obtain confidence intervals. The once-per-series noise-floor pilot (n=3) required by comparison-uncertainty.md is pre-authorized and arrives through the normal repeat_decision: necessary path with a rationale naming the series pilot; it needs no further justification.
• Confirm no other benchmark is targeting the same candidate.
• Every kubectl invocation this skill drives pins --context "${KUBE_CONTEXT}" and -n "${NAMESPACE}", bounds every wait with an explicit timeout, distinguishes job Failed from timeout (poll the job's true condition rather than waiting on Complete alone), and chains commands so a failure never flows into the next step — the same discipline deploy-dynamo-recipe's scripted blocks encode.
• Record neighbour occupancy in benchmark_execution.json at run start and end (the pods scheduled on the serving nodes, e.g. kubectl --context "${KUBE_CONTEXT}" get pods -A -o wide filtered to those nodes), so comparison-uncertainty.md's like-for-like and transition checks have a recorded condition to read.

Execute And Monitor

1. Apply the run-scoped Job.
2. Wait with a bounded timeout and coarse polling.
3. Monitor Job conditions, pod phase/restarts, recent events, and the relevant AIPerf log tail.
4. On completion, copy the complete AIPerf output directory unchanged into <DEPLOY_ROOT>/benchmark/raw_aiperf/ before deleting the Job or pod.
5. Record exact commands, Job/pod names, start/end timestamps, configured and actual measurement duration, whether this was the default run or an approved repeat, repeat rationale, AIPerf source/runtime versions, exit status, retry history, and artifact paths in benchmark_execution.json.

Debug By Ownership

Do not reduce request count, remove difficult trace rows, relax SLOs, lower load, switch schedule mode, or alter ISL/OSL to make a failed benchmark finish. Such a change creates a different benchmark plan and comparison series. Do not launch another valid run because a small delta was classified as noise or because confidence intervals are absent.

Output Status

Set benchmark_execution.json.status to completed, failed, or blocked. A completed Job is not automatically a valid benchmark; validity belongs to the audit phase of analyze-aiperf-results. Record the exact benchmark-plan path, SHA256, series ID, and performance question in the execution record.

Keep logs only when they explain a failure beyond the concise execution ledger. Do not retain routine successful pod logs or broad cluster snapshots.