Aiperf Kube Run

Running an AIPerf Benchmark on Kubernetes

Hub skill for the aiperf kube lifecycle. One AIPerfJob = one benchmark = one controller pod + N worker pods, orchestrated by the in-cluster operator.

Related skills: aiperf-kube-setup (cluster/operator install), aiperf-kube-triage (a run is stuck or failed), aiperf-kube-sweep (multi-variation runs).

Lifecycle

Full lifecycle narrative, phase table, exit codes, and per-command flag reference: references/lifecycle.md (bundled with this skill).

Minimum viable run

aiperf kube preflight -i aiperf:latest -e http://server:8000 -w 1
aiperf kube profile \
    --model Qwen/Qwen3-0.6B \
    --url http://server:8000 \
    --image aiperf:latest \
    --total-workers 8 \
    --detach
aiperf kube attach          # live progress
aiperf kube list            # phase == Completed?
aiperf kube results         # -> ./artifacts/<name>/

-w on preflight counts worker pods; --total-workers on profile counts worker processes, packed runtime.workersPerPod (default 10) per pod. --total-workers 8 is one pod, so preflight it with -w 1; for 8 worker pods use --total-workers 80 -w 8. A total that is neither <= workersPerPod nor a multiple of it is rejected rather than silently collapsed onto one pod.

The benchmark namespace comes from -n/--namespace, or your kubeconfig context when the flag is omitted; there is no fallback. Operator namespace defaults to aiperf-system (auto-detected by cluster-wide pod-label search).

Rules that bite

• Foreground profile in operator mode hard-times-out at 600 s. Anything longer than ~10 minutes MUST use --detach plus aiperf kube attach. Ctrl+C or the timeout does not stop the cluster-side run. The limit is AIPERF_K8S_WATCH_DEFAULT_TIMEOUT_SECONDS (600, max 86400) if you would rather raise it than detach.
• Non-TTY stdout auto-detaches. CI and piped invocations never block; do not write scripts that assume profile waits.
• --dry-run never contacts the cluster, so the mode it prints is assumed, not detected, and the CR is printed before validation. Pass --no-operator to preview direct mode. Use aiperf kube validate for a real verdict.
• ~/.aiperf/last_kube_benchmark.json is per-user, not per-context. Every command that defaults its target reads it. When working across clusters or namespaces in one shell, always pass the job id positionally plus -n and --kube-context.
• kubectl delete leaves that file stale; aiperf kube delete clears it.
• Results come from the operator PVC by default. --from-pods is required for direct-mode runs, and on the default --all path the controller API is the only tier — there is no kubectl cp fallback unless --summary-only.
• Cancel is not delete, but it is not a safe pause either. aiperf kube cancel patches spec.cancel: true; the CR survives, but the operator deletes the JobSet and sets phase=Cancelled without harvesting, and every completion path short-circuits once cancellation is requested. Results survive only if a harvest already finished. Cancel an in-flight run and there is nothing on the PVC and no pods left for --from-pods — download first, cancel second.
• Ambiguous names are refused, not guessed. When an AIPerfJob and an AIPerfSweep share a name, destructive commands need --kind job|sweep. The refusal prints an error and still exits 0, so a script that only checks $? will believe it cancelled or deleted something.
• cancel and delete only know about CRs, and say nothing useful when there isn't one. Both resolve the target by looking up an AIPerfJob or AIPerfSweep CR by name; a direct-mode (--no-operator) run has no AIPerfJob, so they print "No AIPerfJob or AIPerfSweep named ..." and exit 0 while the JobSet keeps running. Tear down direct-mode runs with kubectl delete jobset <name> (plus its ConfigMap/Role/RoleBinding), not with aiperf kube delete.

Exit-code convention

Use --ignore-not-found on attach/logs in teardown scripts. A target that exists but has nothing to show exits 0 by design.

Operator mode vs direct mode

Operator mode is chosen automatically when the aiperfjobs.aiperf.nvidia.com CRD is present. It owns JobSet/ConfigMap/RBAC lifecycle via ownerReferences and harvests results to a PVC on terminal phase (nothing is harvested mid-run).

Direct mode (--no-operator, or CRD absent) creates the resources itself, has no CR and no PVC, and uses an 8-hour JobSet TTL (vs 5 minutes) so pods survive long enough for aiperf kube results --from-pods. Direct-mode details are in references/lifecycle.md.

--operator skips the cluster-scoped CRD probe on its own — that is how namespace-scoped tenants submit without cluster-wide RBAC. Pair it with an explicit --namespace, since skipping the probe also skips operator discovery.

Common mistakes