Triaging an AIPerf Kubernetes Run
Classify first, then follow the branch. Do not read logs before you know the
phase — the phase determines which logs matter.
Related skills: aiperf-kube-run (lifecycle and exit codes),
aiperf-kube-setup (cluster/operator install problems).
The machine-parseable playbook with full JSON schemas and jq/python snippets is
references/debugging.md (bundled with this skill). This skill is the routing
layer.
Step 1: classify
kubectl get aiperfjob <NAME> -n <NS> -o json | jq '{
phase: .status.phase, subPhase: .status.subPhase,
workers: .status.workers, error: .status.error,
conditions: .status.conditions }'
aiperf kube debug -j <NAME> -n <NS> --verbose
aiperf kube debug reports pod states, recent events, node resources, and a
slice of logs from any pod with problems. -A inspects all namespaces. For a
sweep child, pass --variation <idx> (long form only — -v is --verbose
here) and -t <trial>.
Step 2: branch on status.phase
| Phase / symptom |
Likely cause |
Next command |
Pending > ~60 s |
Unschedulable pods |
kubectl get pods -n <NS> -l aiperf.nvidia.com/job-id=<ID> then read PodScheduled reason |
Queued |
Kueue has not admitted the workload |
kubectl get workloads -n <NS> and read .status.conditions[?(@.type=="QuotaReserved")] |
Initializing > ~120 s |
Image pull, ConfigMap, or ZMQ connection probe |
aiperf kube logs <ID> --container control-plane --tail 50 |
Running, restarts > 3 |
Crash loop |
aiperf kube logs <ID> --container <c> --tail 100 |
Running, OOM-killed pod |
Node memory pressure, not a container limit (see below) |
kubectl describe node for pressure; raise AIPERF_K8S_WORKER_POD_MEMORY / AIPERF_K8S_SYSTEM_CONTROLLER_MEMORY on the operator deployment |
Running, requestsCompleted flat |
Stalled benchmark |
check endpoint reachability from inside the cluster |
Running, request_error_rate.avg > 5 |
Endpoint rejecting requests (the metric is a percent, so 5 means 5%) |
inspect worker logs and server-side errors |
Failed |
See failure table below |
kubectl ... .status.error + controller logs |
Cancelled |
Someone (or spec.cancel) stopped it |
no action |
Completed |
Done |
aiperf kube results |
request_error_rate.avg in status.liveMetrics.metrics is the only error
signal published while a job runs — error_request_count is ERROR_ONLY and is
filtered out of live metrics. Do not conclude "no errors" from its absence.
Step 3: failure patterns
status.error contains |
Root cause |
Fix |
preflight |
Cluster validation failed |
aiperf kube preflight -o json, fix failing checks |
endpoint, health check |
Server unreachable from the cluster |
resolve the URL from inside a pod, not from your laptop |
timeout |
Exceeded spec.timeoutSeconds |
raise it, or set 0 |
ConfigMap, size |
Config over the 1 MiB object limit |
shrink the config / move data to a mounted file |
image, pull |
Image not accessible |
check tag and imagePullSecrets; on Kind, kind load + pullPolicy: Never |
RBAC, forbidden |
Missing permissions |
check service account, Role, RoleBinding |
Scheduling messages map directly to causes: Insufficient cpu/memory -> lower
--total-workers; nvidia.com/gpu -> no free GPU nodes; didn't match Pod's node affinity/selector -> wrong spec.podTemplate.nodeSelector; had untolerated taint -> missing spec.podTemplate.tolerations; quota ->
namespace ResourceQuota exhausted.
None of those are patchable on the live CR. Nineteen spec fields are
immutable after creation — image, imagePullPolicy, benchmark,
resourceMode, connectionsPerWorker, ttlSecondsAfterFinished,
resultsTtlDays, keepFailedPods, podTemplate, scheduling,
schemaVersion, sweep, multiRun, plot, variables, randomSeed,
noSweepTable, skipEndpointCheck, failurePolicy. Only timeoutSeconds and
cancel accept an update; everything else rejects the patch with
spec.<field> is immutable after creation. So every scheduling, placement,
resource, priority, or workload fix means delete and resubmit, not
kubectl edit.
Rules that bite
- A benchmark that looks hung at high concurrency is often CPU starvation of a
single service, not a deadlock. The records-manager and system-controller
default to
75m CPU (AIPERF_K8S_RECORDS_MANAGER_CPU,
AIPERF_K8S_SYSTEM_CONTROLLER_CPU). At very high request rates a pegged core
starves the event loop, heartbeats expire, and the CR freezes mid-phase. Raise
the CPU before hunting for a logic bug.
- The default
resourceMode is burstable, so pods have requests and no
limits. spec.resourceMode defaults to burstable
and emits requests only. That means
a container cannot be cgroup-OOM-killed for exceeding its memory budget and
cannot be CFS-throttled for exceeding its CPU budget — the 75m/150m
numbers are scheduling hints, not ceilings. Two consequences: an OOMKilled
container is evidence of node memory pressure or kubelet eviction, not of a
too-small AIPERF_K8S_*_MEMORY; and a starving service is losing a fair-share
race against co-tenants, which raising the request fixes only indirectly (by
moving the pod or reserving more share). Set resourceMode: guaranteed if you
actually want the env budgets enforced as limits, and expect real OOM kills
once you do.
- Heartbeat expiry is a symptom, not a cause.
AIPERF_K8S_CONTROLLER_HEARTBEAT_EXPIRY_SECONDS
defaults to 30 s and must be at least twice the interval; widening it hides
the starvation instead of fixing it.
- Workers pass two startup probes, and they fail differently. The PUB/SUB
connection probe fails closed: the pod exits on purpose so Kubernetes
restarts it with a fresh ZMQ context
(
AIPERF_K8S_JOBSET_WORKER_CONNECTION_PROBE_TIMEOUT, backoff limit 20). A
few early restarts on first deploy are expected, not a bug. The credit-return
PUSH probe fails open: credit dispatch rides the DEALER but returns ride a
separate PUSH/PULL fan-in, and on budget expiry
(AIPERF_WORKER_RETURN_PROBE_BUDGET, default 30 s;
AIPERF_WORKER_RETURN_PROBE_RETRY_DELAY, default 0.1 s) the worker announces
dispatchability anyway and logs a warning. So a dead return path produces a
stalled run with zero restarts -- grep worker logs for
Credit-return channel still has no peer before blaming the endpoint.
- Operator-mode results are harvested only on terminal phase. "results
missing" mid-run is correct behavior; use
--from-pods to read the live
controller instead.
- On a successful run the pods do not last 300 s — they are deleted at
once.
AIPERF_K8S_JOBSET_TTL_SECONDS_AFTER_FINISHED (300 s) is only the
fallback: once the operator has harvested results it calls
its post-success cleanup and deletes the JobSet immediately. The TTL window is real only for failed runs and for partial
harvests that are being retried. Never plan to "grab the logs after it
finishes" — stream them with --follow during the run, or accept that a
clean success leaves you only the PVC artifacts. Direct (operator-less) mode
is the opposite: AIPERF_K8S_JOBSET_DIRECT_MODE_TTL_SECONDS defaults to
28800 s (8 h) so pods stay around for manual retrieval.
debug, cancel, delete, list exit 0 on a missing target by design.
Never gate a script on their exit status; use attach or logs.
- First deploy attempts flake more than steady state (ConfigMap propagation,
ZMQ probe). One clean retry before deep investigation is a legitimate step.
Diagnosis thresholds
aiperf kube debug findings are tunable via AIPERF_K8S_DIAGNOSIS_*:
STALLED_PENDING_THRESHOLD_SECONDS (60), STALLED_RUNNING_THRESHOLD_SECONDS
(30), HIGH_ERROR_RATE_THRESHOLD (0.05), FAIL_ABOVE_ERROR_RATE (1.0),
HIGH_LATENCY_P99_MULTIPLIER (10). Full env-var table in
references/debugging.md.
Log collection
aiperf kube logs <ID> -n <NS> -o ./triage --ignore-not-found # per-pod files
aiperf kube logs <ID> --container control-plane --follow
Containers worth naming: control-plane (SystemController), api,
event-bus-proxy, worker containers. Save logs to a directory before the TTL
reaps the pods.
1---2name: aiperf-kube-triage3description: Use when an AIPerf benchmark on Kubernetes is stuck, failing, crash-looping, OOM-killed, pending, queued, stalled, showing a high error rate, or its results cannot be retrieved - diagnosing an AIPerfJob or AIPerfSweep from CR status, pod state, and logs.4metadata:5 source: https://skillmd.com/skills/ai-dynamo/aiperf-kube-triage6---78# Triaging an AIPerf Kubernetes Run910Classify first, then follow the branch. Do not read logs before you know the11phase — the phase determines which logs matter.1213**Related skills:** `aiperf-kube-run` (lifecycle and exit codes),14`aiperf-kube-setup` (cluster/operator install problems).1516The machine-parseable playbook with full JSON schemas and jq/python snippets is17`references/debugging.md` (bundled with this skill). This skill is the routing18layer.1920## Step 1: classify2122```bash23kubectl get aiperfjob <NAME> -n <NS> -o json | jq '{24 phase: .status.phase, subPhase: .status.subPhase,25 workers: .status.workers, error: .status.error,26 conditions: .status.conditions }'2728aiperf kube debug -j <NAME> -n <NS> --verbose29```3031`aiperf kube debug` reports pod states, recent events, node resources, and a32slice of logs from any pod with problems. `-A` inspects all namespaces. For a33sweep child, pass `--variation <idx>` (long form only — `-v` is `--verbose`34here) and `-t <trial>`.3536## Step 2: branch on `status.phase`3738| Phase / symptom | Likely cause | Next command |39|---|---|---|40| `Pending` > ~60 s | Unschedulable pods | `kubectl get pods -n <NS> -l aiperf.nvidia.com/job-id=<ID>` then read `PodScheduled` reason |41| `Queued` | Kueue has not admitted the workload | `kubectl get workloads -n <NS>` and read `.status.conditions[?(@.type=="QuotaReserved")]` |42| `Initializing` > ~120 s | Image pull, ConfigMap, or ZMQ connection probe | `aiperf kube logs <ID> --container control-plane --tail 50` |43| `Running`, restarts > 3 | Crash loop | `aiperf kube logs <ID> --container <c> --tail 100` |44| `Running`, OOM-killed pod | Node memory pressure, *not* a container limit (see below) | `kubectl describe node` for pressure; raise `AIPERF_K8S_WORKER_POD_MEMORY` / `AIPERF_K8S_SYSTEM_CONTROLLER_MEMORY` on the **operator** deployment |45| `Running`, `requestsCompleted` flat | Stalled benchmark | check endpoint reachability from inside the cluster |46| `Running`, `request_error_rate.avg > 5` | Endpoint rejecting requests (the metric is a **percent**, so 5 means 5%) | inspect worker logs and server-side errors |47| `Failed` | See failure table below | `kubectl ... .status.error` + controller logs |48| `Cancelled` | Someone (or `spec.cancel`) stopped it | no action |49| `Completed` | Done | `aiperf kube results` |5051`request_error_rate.avg` in `status.liveMetrics.metrics` is the **only** error52signal published while a job runs — `error_request_count` is `ERROR_ONLY` and is53filtered out of live metrics. Do not conclude "no errors" from its absence.5455## Step 3: failure patterns5657| `status.error` contains | Root cause | Fix |58|---|---|---|59| `preflight` | Cluster validation failed | `aiperf kube preflight -o json`, fix failing checks |60| `endpoint`, `health check` | Server unreachable from the cluster | resolve the URL from inside a pod, not from your laptop |61| `timeout` | Exceeded `spec.timeoutSeconds` | raise it, or set `0` |62| `ConfigMap`, `size` | Config over the 1 MiB object limit | shrink the config / move data to a mounted file |63| `image`, `pull` | Image not accessible | check tag and `imagePullSecrets`; on Kind, `kind load` + `pullPolicy: Never` |64| `RBAC`, `forbidden` | Missing permissions | check service account, Role, RoleBinding |6566Scheduling messages map directly to causes: `Insufficient cpu/memory` -> lower67`--total-workers`; `nvidia.com/gpu` -> no free GPU nodes; `didn't match Pod's68node affinity/selector` -> wrong `spec.podTemplate.nodeSelector`; `had69untolerated taint` -> missing `spec.podTemplate.tolerations`; `quota` ->70namespace ResourceQuota exhausted.7172**None of those are patchable on the live CR.** Nineteen `spec` fields are73immutable after creation — `image`, `imagePullPolicy`, `benchmark`,74`resourceMode`, `connectionsPerWorker`, `ttlSecondsAfterFinished`,75`resultsTtlDays`, `keepFailedPods`, `podTemplate`, `scheduling`,76`schemaVersion`, `sweep`, `multiRun`, `plot`, `variables`, `randomSeed`,77`noSweepTable`, `skipEndpointCheck`, `failurePolicy`. Only `timeoutSeconds` and78`cancel` accept an update; everything else rejects the patch with79`spec.<field> is immutable after creation`. So every scheduling, placement,80resource, priority, or workload fix means **delete and resubmit**, not81`kubectl edit`.8283## Rules that bite8485- **A benchmark that looks hung at high concurrency is often CPU starvation of a86 single service, not a deadlock.** The records-manager and system-controller87 default to `75m` CPU (`AIPERF_K8S_RECORDS_MANAGER_CPU`,88 `AIPERF_K8S_SYSTEM_CONTROLLER_CPU`). At very high request rates a pegged core89 starves the event loop, heartbeats expire, and the CR freezes mid-phase. Raise90 the CPU before hunting for a logic bug.91- **The default `resourceMode` is `burstable`, so pods have requests and no92 limits.** `spec.resourceMode` defaults to `burstable`93 and emits `requests` only. That means94 a container cannot be cgroup-OOM-killed for exceeding its memory budget and95 cannot be CFS-throttled for exceeding its CPU budget — the `75m`/`150m`96 numbers are scheduling hints, not ceilings. Two consequences: an `OOMKilled`97 container is evidence of *node* memory pressure or kubelet eviction, not of a98 too-small `AIPERF_K8S_*_MEMORY`; and a starving service is losing a fair-share99 race against co-tenants, which raising the request fixes only indirectly (by100 moving the pod or reserving more share). Set `resourceMode: guaranteed` if you101 actually want the env budgets enforced as limits, and expect real OOM kills102 once you do.103- **Heartbeat expiry is a symptom, not a cause.** `AIPERF_K8S_CONTROLLER_HEARTBEAT_EXPIRY_SECONDS`104 defaults to 30 s and must be at least twice the interval; widening it hides105 the starvation instead of fixing it.106- **Workers pass two startup probes, and they fail differently.** The PUB/SUB107 connection probe fails *closed*: the pod exits on purpose so Kubernetes108 restarts it with a fresh ZMQ context109 (`AIPERF_K8S_JOBSET_WORKER_CONNECTION_PROBE_TIMEOUT`, backoff limit 20). A110 few early restarts on first deploy are expected, not a bug. The credit-return111 PUSH probe fails *open*: credit dispatch rides the DEALER but returns ride a112 separate PUSH/PULL fan-in, and on budget expiry113 (`AIPERF_WORKER_RETURN_PROBE_BUDGET`, default 30 s;114 `AIPERF_WORKER_RETURN_PROBE_RETRY_DELAY`, default 0.1 s) the worker announces115 dispatchability anyway and logs a warning. So a dead return path produces a116 stalled run with **zero restarts** -- grep worker logs for117 `Credit-return channel still has no peer` before blaming the endpoint.118- **Operator-mode results are harvested only on terminal phase.** "results119 missing" mid-run is correct behavior; use `--from-pods` to read the live120 controller instead.121- **On a successful run the pods do not last 300 s — they are deleted at122 once.** `AIPERF_K8S_JOBSET_TTL_SECONDS_AFTER_FINISHED` (300 s) is only the123 fallback: once the operator has harvested results it calls124 its post-success cleanup and deletes the JobSet immediately. The TTL window is real only for failed runs and for partial125 harvests that are being retried. Never plan to "grab the logs after it126 finishes" — stream them with `--follow` during the run, or accept that a127 clean success leaves you only the PVC artifacts. Direct (operator-less) mode128 is the opposite: `AIPERF_K8S_JOBSET_DIRECT_MODE_TTL_SECONDS` defaults to129 28800 s (8 h) so pods stay around for manual retrieval.130- **`debug`, `cancel`, `delete`, `list` exit 0 on a missing target** by design.131 Never gate a script on their exit status; use `attach` or `logs`.132- **First deploy attempts flake more than steady state** (ConfigMap propagation,133 ZMQ probe). One clean retry before deep investigation is a legitimate step.134135## Diagnosis thresholds136137`aiperf kube debug` findings are tunable via `AIPERF_K8S_DIAGNOSIS_*`:138`STALLED_PENDING_THRESHOLD_SECONDS` (60), `STALLED_RUNNING_THRESHOLD_SECONDS`139(30), `HIGH_ERROR_RATE_THRESHOLD` (0.05), `FAIL_ABOVE_ERROR_RATE` (1.0),140`HIGH_LATENCY_P99_MULTIPLIER` (10). Full env-var table in141`references/debugging.md`.142143## Log collection144145```bash146aiperf kube logs <ID> -n <NS> -o ./triage --ignore-not-found # per-pod files147aiperf kube logs <ID> --container control-plane --follow148```149150Containers worth naming: `control-plane` (SystemController), `api`,151`event-bus-proxy`, worker containers. Save logs to a directory before the TTL152reaps the pods.