# Aks Validate

> Generates a read-only validation script that confirms an AKS cluster bootstrapped via iam-vpc → aks-cluster → aks-nodes → acr → blob-storage → gpu-operator is healthy end-to-end. Checks RG, VNet, AKS provisioning state, node pool composition, ACR + AcrPull role, Storage + Blob Data Contributor role, GPU operator pods, GPU resource advertisement, a CUDA smoke pod, and a Blob auth probe. Emits PASS/WARN/FAIL plus a JSON report. Emitted as Stage 07 by `aks-bootstrap`; can be run standalone after the stack is up. Trigger when the user mentions "validate the cluster", "is the cluster ready", "smoke test the bootstrap", "kick the tires", or after any stack change.

- Skill: `nvidia-omniverse/aks-validate` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nvidia-omniverse/aks-validate`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-omniverse/aks-validate/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: NVIDIA-Omniverse (https://skillmd.com/u/nvidia-omniverse)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia-omniverse/aks-validate

---


<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: CC-BY-4.0 AND Apache-2.0 -->

# AKS bootstrap validation

## What this skill produces

A read-only bash script that runs ~10 checks against the live stack and emits:

- Stdout: per-check `PASS`, `WARN`, `FAIL` lines + summary.
- `generated/stage07-<cluster-name>-validate-report.json` — machine-readable report.
- Exit code: `0` if all checks PASS (WARN is non-fatal), `1` if any FAIL, `2` if preflight failed (missing tool, cluster unreachable).

This is the "cluster is ready" gate. Tolerant in orchestrator mode (see `aks-bootstrap`).

## Inputs

Source from `streaming-env.sh` and any `.env.*` files written by upstream stages.

**Required:**

| Variable | Description |
|---|---|
| `CLUSTER_NAME` | Name |
| `RESOURCE_GROUP` | RG |
| `AZURE_SUBSCRIPTION_ID` | Subscription |

**Optional (passed through from upstream stages for observed-vs-expected checks):**

| Variable | Description | Default |
|---|---|---|
| `GPU_POOL_NAME`, `GPU_VM_SIZE`, `GPU_MIN_COUNT`, `GPU_MAX_COUNT` | from `aks-nodes` | (unset) |
| `ACR_NAME`, `KUBELET_IDENTITY_OBJECT_ID` | from `acr`/`aks-cluster` | (unset) |
| `STORAGE_ACCOUNT_NAME` | from `blob-storage` | (unset) |
| `BLOB_PROBE_NAMESPACE` | Namespace containing the workload identity service account | `default` |
| `BLOB_PROBE_SERVICE_ACCOUNT` | K8s ServiceAccount bound to a workload identity with Blob access | (unset; skips probe with WARN) |
| `BLOB_PROBE_IMAGE` | Azure CLI image used for the in-cluster Blob auth probe | `mcr.microsoft.com/azure-cli:2.61.0` |
| `SKIP_CUDA` | Skip the CUDA smoke pod | `false` |
| `SKIP_BLOB_PROBE` | Skip the Blob auth probe | `false` |
| `TIMEOUT_MIN` | Per-check timeout cap | `5` |

## Output

`generated/stage07-<cluster-name>-validate.sh` and at runtime, `generated/stage07-<cluster-name>-validate-report.json`.

JSON shape:

```json
{
  "cluster": "aks-nvcf-validation-westus3",
  "timestamp": "2026-05-27T12:34:56Z",
  "summary": { "pass": 8, "warn": 1, "fail": 0 },
  "checks": [
    { "id": "rg",          "status": "PASS", "detail": "RG exists in westus3" },
    { "id": "vnet",        "status": "PASS", "detail": "VNet 10.10.0.0/16; subnets: snet-aks-nodes, snet-aks-api-server" },
    { "id": "aks",         "status": "PASS", "detail": "Succeeded; v1.30.2" },
    { "id": "nodes",       "status": "PASS", "detail": "system=1, gpu=1 (Standard_NV36ads_A10_v5, zone 1)" },
    { "id": "acr-pull",    "status": "PASS", "detail": "AcrPull granted to kubelet identity" },
    { "id": "blob-rbac",   "status": "PASS", "detail": "Storage Blob Data Contributor on grantee" },
    { "id": "gpu-op",      "status": "PASS", "detail": "9/9 pods Ready; chart v25.3.0" },
    { "id": "gpu-resource","status": "PASS", "detail": "1 node advertises nvidia.com/gpu=1" },
    { "id": "cuda-smoke",  "status": "PASS", "detail": "nvidia-smi pod returned 0; A10 detected" },
    { "id": "blob-probe",  "status": "WARN", "detail": "Skipped (no STORAGE_ACCOUNT_NAME)" }
  ]
}
```

## Checks

| ID | What it verifies | Severity if fail |
|---|---|---|
| `rg` | RG exists in the expected region | FAIL |
| `vnet` | VNet exists; node + API-server subnets present | FAIL |
| `aks` | `provisioningState=Succeeded`; OIDC issuer URL non-empty; workload identity enabled | FAIL |
| `nodes` | Expected pools present; `node-vm-size` matches `aks-nodes` inputs; GPU pool zone matches | FAIL |
| `acr-pull` | `AcrPull` role assignment exists on the ACR scope to the kubelet identity | FAIL |
| `blob-rbac` | `Storage Blob Data Contributor` on the storage account scope to the grantee | WARN if grantee/account unset |
| `gpu-op` | GPU operator pods all Ready; chart version matches `GPU_OPERATOR_VERSION` if set | FAIL |
| `gpu-resource` | At least one node advertises `nvidia.com/gpu` >= 1 | FAIL |
| `cuda-smoke` | Pod with `nvcr.io/nvidia/cuda:...` requesting 1 GPU runs `nvidia-smi`, exits 0; pod is cleaned up | FAIL unless `SKIP_CUDA=true` |
| `blob-probe` | (Optional) From a pod using `BLOB_PROBE_SERVICE_ACCOUNT`, `az storage blob list --auth-mode login` succeeds | WARN if `SKIP_BLOB_PROBE=true`, storage account unset, or service account unset |

Each check has a per-check timeout (default 60s; cuda-smoke up to 5min).

## Bash script (complete)

```bash
#!/bin/bash
set -uo pipefail   # NOT -e — we want to collect all check results, not bail on the first fail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
for envfile in .env.iam-vpc .env.aks-cluster .env.acr .env.blob-storage; do
  [[ -f "$REPO_ROOT/generated/$envfile" ]] && source "$REPO_ROOT/generated/$envfile"
done

: "${CLUSTER_NAME:?}" "${RESOURCE_GROUP:?}" "${AZURE_SUBSCRIPTION_ID:?}"
export KUBECONFIG="${KUBECONFIG:-/tmp/kubeconfig-${CLUSTER_NAME}}"
SKIP_CUDA="${SKIP_CUDA:-false}"
SKIP_BLOB_PROBE="${SKIP_BLOB_PROBE:-false}"
TIMEOUT_MIN="${TIMEOUT_MIN:-5}"

mkdir -p "$REPO_ROOT/generated"
REPORT="$REPO_ROOT/generated/stage07-${CLUSTER_NAME}-validate-report.json"
declare -a CHECKS
PASS=0; WARN=0; FAIL=0

record() {
  local id="$1" status="$2" detail="$3"
  CHECKS+=("$(printf '{"id":"%s","status":"%s","detail":"%s"}' "$id" "$status" "$(echo "$detail" | sed 's/"/\\"/g')")")
  case "$status" in PASS) ((PASS++));; WARN) ((WARN++));; FAIL) ((FAIL++));; esac
  printf "[%s] %s — %s\n" "$status" "$id" "$detail"
}

# Preflight
for tool in az kubectl jq; do
  command -v "$tool" >/dev/null || { echo "MISSING TOOL: $tool"; exit 2; }
done
az account show >/dev/null 2>&1 || { echo "az not authenticated"; exit 2; }
kubectl cluster-info >/dev/null 2>&1 || { echo "kubectl can't reach cluster $CLUSTER_NAME"; exit 2; }

# --- check: rg ---
LOC=$(az group show -n "$RESOURCE_GROUP" --query location -o tsv 2>/dev/null || echo "")
if [[ -n "$LOC" ]]; then record rg PASS "RG exists in $LOC"; else record rg FAIL "RG $RESOURCE_GROUP not found"; fi

# --- check: vnet ---
VNET_NAME="${VNET_NAME:-}"
NODE_SUBNET_NAME="${NODE_SUBNET_NAME:-snet-aks-nodes}"
API_SERVER_SUBNET_NAME="${API_SERVER_SUBNET_NAME:-snet-aks-api-server}"
if [[ -z "$VNET_NAME" ]]; then
  record vnet WARN "VNET_NAME unset; skipping VNet check"
else
  VNET_EXISTS=$(az network vnet show -n "$VNET_NAME" -g "$RESOURCE_GROUP" --query addressSpace.addressPrefixes[0] -o tsv 2>/dev/null || echo "")
  if [[ -z "$VNET_EXISTS" ]]; then
    record vnet FAIL "VNet $VNET_NAME not found in $RESOURCE_GROUP"
  else
    SUBNETS=$(az network vnet subnet list --vnet-name "$VNET_NAME" -g "$RESOURCE_GROUP" --query "[].name" -o tsv 2>/dev/null || echo "")
    MISSING=""
    [[ "$SUBNETS" != *"$NODE_SUBNET_NAME"* ]] && MISSING="$NODE_SUBNET_NAME"
    [[ "$SUBNETS" != *"$API_SERVER_SUBNET_NAME"* ]] && MISSING="${MISSING:+$MISSING, }$API_SERVER_SUBNET_NAME"
    if [[ -n "$MISSING" ]]; then
      record vnet FAIL "VNet $VNET_NAME exists ($VNET_EXISTS) but missing subnets: $MISSING"
    else
      record vnet PASS "VNet $VNET_NAME ($VNET_EXISTS); subnets: $NODE_SUBNET_NAME, $API_SERVER_SUBNET_NAME"
    fi
  fi
fi

# --- check: aks ---
STATE=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query provisioningState -o tsv 2>/dev/null || echo "")
OIDC=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query oidcIssuerProfile.issuerUrl -o tsv 2>/dev/null || echo "")
WI=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query securityProfile.workloadIdentity.enabled -o tsv 2>/dev/null || echo "")
K8S_VER=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query currentKubernetesVersion -o tsv 2>/dev/null || echo "")
if [[ "$STATE" == "Succeeded" && -n "$OIDC" && "$WI" == "true" ]]; then
  record aks PASS "Succeeded; v${K8S_VER}; OIDC + workload identity enabled"
else
  record aks FAIL "state=$STATE oidc=${OIDC:-unset} wi=${WI:-unset}"
fi

# --- check: nodes ---
GPU_POOL_NAME="${GPU_POOL_NAME:-}"
GPU_VM_SIZE="${GPU_VM_SIZE:-}"
GPU_POOL_ZONE="${GPU_POOL_ZONE:-}"
POOL_JSON=$(az aks nodepool list -g "$RESOURCE_GROUP" --cluster-name "$CLUSTER_NAME" -o json 2>/dev/null || echo "[]")
SYSTEM_COUNT=$(echo "$POOL_JSON" | jq '[.[] | select(.mode=="System")] | length')
GPU_COUNT=$(echo "$POOL_JSON" | jq --arg n "${GPU_POOL_NAME:-gpu}" '[.[] | select(.name==$n or .mode=="User")] | length')
if [[ "$SYSTEM_COUNT" -eq 0 ]]; then
  record nodes FAIL "No system node pool found"
elif [[ "$GPU_COUNT" -eq 0 ]]; then
  record nodes FAIL "No user/GPU node pool found (expected name=${GPU_POOL_NAME:-gpu})"
else
  DETAIL="system=${SYSTEM_COUNT}, gpu=${GPU_COUNT}"
  NODE_STATUS="PASS"
  if [[ -n "$GPU_POOL_NAME" && -n "$GPU_VM_SIZE" ]]; then
    ACTUAL_SIZE=$(echo "$POOL_JSON" | jq -r --arg n "$GPU_POOL_NAME" '.[] | select(.name==$n) | .vmSize // ""')
    if [[ -n "$ACTUAL_SIZE" && "$ACTUAL_SIZE" != "$GPU_VM_SIZE" ]]; then
      DETAIL="$DETAIL; vm-size mismatch: expected=$GPU_VM_SIZE actual=$ACTUAL_SIZE"
      NODE_STATUS="FAIL"
    else
      DETAIL="$DETAIL; vm-size=$ACTUAL_SIZE"
    fi
  fi
  if [[ -n "$GPU_POOL_NAME" && -n "$GPU_POOL_ZONE" ]]; then
    ACTUAL_ZONES=$(echo "$POOL_JSON" | jq -r --arg n "$GPU_POOL_NAME" '.[] | select(.name==$n) | (.availabilityZones // []) | join(",")')
    DETAIL="$DETAIL; zone=${ACTUAL_ZONES:-unset}"
  fi
  record nodes "$NODE_STATUS" "$DETAIL"
fi

# --- check: acr-pull ---
if [[ -z "${ACR_NAME:-}" || -z "${KUBELET_IDENTITY_OBJECT_ID:-}" ]]; then
  record acr-pull WARN "ACR_NAME or KUBELET_IDENTITY_OBJECT_ID unset; skipping AcrPull check"
else
  ACR_ID=$(az acr show -n "$ACR_NAME" -g "$RESOURCE_GROUP" --query id -o tsv 2>/dev/null || echo "")
  if [[ -z "$ACR_ID" ]]; then
    record acr-pull FAIL "ACR $ACR_NAME not found in $RESOURCE_GROUP"
  else
    ROLE_COUNT=$(az role assignment list --scope "$ACR_ID" \
      --query "[?principalId=='${KUBELET_IDENTITY_OBJECT_ID}' && roleDefinitionName=='AcrPull'] | length(@)" \
      -o tsv 2>/dev/null || echo "0")
    if [[ "${ROLE_COUNT:-0}" -gt 0 ]]; then
      record acr-pull PASS "AcrPull granted to kubelet identity on $ACR_NAME"
    else
      record acr-pull FAIL "AcrPull not found on $ACR_NAME for kubelet identity $KUBELET_IDENTITY_OBJECT_ID"
    fi
  fi
fi

# --- check: blob-rbac ---
if [[ -z "${STORAGE_ACCOUNT_NAME:-}" ]]; then
  record blob-rbac WARN "STORAGE_ACCOUNT_NAME unset; skipping Blob RBAC check"
else
  SA_ID=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP" --query id -o tsv 2>/dev/null || echo "")
  if [[ -z "$SA_ID" ]]; then
    record blob-rbac FAIL "Storage account $STORAGE_ACCOUNT_NAME not found"
  else
    BLOB_ROLE_COUNT=$(az role assignment list --scope "$SA_ID" \
      --query "[?roleDefinitionName=='Storage Blob Data Contributor'] | length(@)" \
      -o tsv 2>/dev/null || echo "0")
    if [[ "${BLOB_ROLE_COUNT:-0}" -gt 0 ]]; then
      record blob-rbac PASS "Storage Blob Data Contributor assigned on $STORAGE_ACCOUNT_NAME"
    else
      record blob-rbac FAIL "Storage Blob Data Contributor not found on $STORAGE_ACCOUNT_NAME"
    fi
  fi
fi

# --- check: gpu-op ---
GPU_OP_NS="gpu-operator"
GPU_OP_TIMEOUT="${TIMEOUT_MIN}m"
PODS_JSON=$(kubectl get pods -n "$GPU_OP_NS" -o json 2>/dev/null || echo '{"items":[]}')
TOTAL=$(echo "$PODS_JSON" | jq '.items | length')
if [[ "$TOTAL" -eq 0 ]]; then
  record gpu-op FAIL "No pods found in namespace $GPU_OP_NS"
else
  NOT_READY=$(echo "$PODS_JSON" | jq '[.items[] | select(.status.phase != "Succeeded") | select(((.status.conditions // []) | map(select(.type=="Ready" and .status=="True")) | length) == 0)] | length')
  CHART_VER=$(kubectl get helmrelease -n "$GPU_OP_NS" gpu-operator -o jsonpath='{.spec.chart.spec.version}' 2>/dev/null || echo "")
  DETAIL="${TOTAL} pods; ready-condition-missing=${NOT_READY}${CHART_VER:+; chart=$CHART_VER}"
  if [[ "$NOT_READY" -eq 0 ]]; then
    record gpu-op PASS "$DETAIL"
  else
    record gpu-op FAIL "$DETAIL"
  fi
fi

# --- check: gpu-resource ---
GPU_NODE_COUNT=$(kubectl get nodes -o json 2>/dev/null \
  | jq '[.items[] | select((.status.capacity["nvidia.com/gpu"] // "0" | tonumber) >= 1)] | length')
if [[ "${GPU_NODE_COUNT:-0}" -gt 0 ]]; then
  record gpu-resource PASS "${GPU_NODE_COUNT} node(s) advertise nvidia.com/gpu >= 1"
else
  record gpu-resource FAIL "No nodes advertise nvidia.com/gpu (GPU operator may not be ready)"
fi

# --- check: cuda-smoke ---
if [[ "$SKIP_CUDA" == "true" ]]; then
  record cuda-smoke WARN "Skipped (SKIP_CUDA=true)"
else
  SMOKE_POD="validate-cuda-smoke-$$"
  CUDA_IMAGE="${CUDA_IMAGE:-nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04}"
  GPU_TAINT_KEY="${GPU_TAINT_KEY:-nvidia.com/gpu}"
  # Ensure pod is deleted on every exit path from this check
  cleanup_smoke() { kubectl delete pod "$SMOKE_POD" --ignore-not-found >/dev/null 2>&1 || true; }
  trap cleanup_smoke EXIT
  # Schedule onto a GPU node via the resource limit alone (no node-type selector —
  # the GPU pool is labeled nvidia.com/gpu=true, never node-type=gpu). The toleration
  # is harmless when the pool has no taint (repo default) and works if GPU_NODE_TAINTS is set.
  cat <<EOF | kubectl apply -f - >/dev/null 2>&1
apiVersion: v1
kind: Pod
metadata: { name: $SMOKE_POD, namespace: default }
spec:
  restartPolicy: Never
  tolerations:
    - { key: "$GPU_TAINT_KEY", operator: Exists, effect: NoSchedule }
  containers:
    - name: nvidia-smi
      image: $CUDA_IMAGE
      command: ["nvidia-smi"]
      resources: { limits: { nvidia.com/gpu: 1 } }
EOF
  if kubectl wait pod "$SMOKE_POD" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${TIMEOUT_MIN}m" >/dev/null 2>&1; then
    GPU_MODEL=$(kubectl logs "$SMOKE_POD" 2>/dev/null | grep -m1 "GPU" | sed 's/^[[:space:]]*//' || echo "unknown")
    record cuda-smoke PASS "nvidia-smi exited 0; ${GPU_MODEL}"
  else
    POD_PHASE=$(kubectl get pod "$SMOKE_POD" -o jsonpath='{.status.phase}' 2>/dev/null || echo "unknown")
    record cuda-smoke FAIL "nvidia-smi pod did not complete (phase=$POD_PHASE)"
  fi
  cleanup_smoke
  trap - EXIT
fi

# --- check: blob-probe ---
if [[ "$SKIP_BLOB_PROBE" == "true" || -z "${STORAGE_ACCOUNT_NAME:-}" || -z "${BLOB_PROBE_SERVICE_ACCOUNT:-}" ]]; then
  record blob-probe WARN "Skipped (SKIP_BLOB_PROBE=${SKIP_BLOB_PROBE} STORAGE_ACCOUNT_NAME=${STORAGE_ACCOUNT_NAME:-unset} BLOB_PROBE_SERVICE_ACCOUNT=${BLOB_PROBE_SERVICE_ACCOUNT:-unset})"
else
  BLOB_PROBE_NAMESPACE="${BLOB_PROBE_NAMESPACE:-default}"
  BLOB_PROBE_IMAGE="${BLOB_PROBE_IMAGE:-mcr.microsoft.com/azure-cli:2.61.0}"
  BLOB_CONTAINER="$(echo "${STORAGE_CONTAINERS:-scratch}" | cut -d, -f1)"
  BLOB_POD="validate-blob-probe-$$"
  cleanup_blob_probe() { kubectl delete pod "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true; }
  trap cleanup_blob_probe EXIT
  kubectl run "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" \
    --image="$BLOB_PROBE_IMAGE" \
    --restart=Never \
    --overrides="{\"spec\":{\"serviceAccountName\":\"${BLOB_PROBE_SERVICE_ACCOUNT}\"}}" \
    -- /bin/sh -lc "az storage blob list --account-name '${STORAGE_ACCOUNT_NAME}' --container-name '${BLOB_CONTAINER}' --auth-mode login --output none" >/dev/null 2>&1
  if kubectl wait pod "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${TIMEOUT_MIN}m" >/dev/null 2>&1; then
    record blob-probe PASS "workload identity pod listed blobs in $STORAGE_ACCOUNT_NAME/$BLOB_CONTAINER using serviceAccount=$BLOB_PROBE_SERVICE_ACCOUNT"
  else
    POD_PHASE=$(kubectl get pod "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" -o jsonpath='{.status.phase}' 2>/dev/null || echo "unknown")
    record blob-probe FAIL "workload identity Blob probe failed (phase=$POD_PHASE serviceAccount=$BLOB_PROBE_SERVICE_ACCOUNT) — check federated credential / RBAC"
  fi
  cleanup_blob_probe
  trap - EXIT
fi

# --- emit report ---
printf '{ "cluster":"%s","timestamp":"%s","summary":{"pass":%d,"warn":%d,"fail":%d},"checks":[%s] }\n' \
  "$CLUSTER_NAME" "$(date -u +%FT%TZ)" "$PASS" "$WARN" "$FAIL" \
  "$(IFS=,; echo "${CHECKS[*]}")" | jq . > "$REPORT"

echo
echo "=== Summary: PASS=$PASS WARN=$WARN FAIL=$FAIL ==="
echo "Report: $REPORT"
[[ "$FAIL" -gt 0 ]] && exit 1 || exit 0
```

## Tolerant execution

When run from `aks-bootstrap` deploy mode, the orchestrator reports the exit code but does NOT treat exit 1 as a bootstrap failure — stages 01–06 have already succeeded. The validate stage is diagnostic, not constructive. See `aks-bootstrap` for the contract.

## Validation checklist (for the generator)

- [ ] Script uses `set -uo pipefail` (NOT `-e` — needs to keep collecting results past failures)
- [ ] Preflight exits with code 2 (not 1) if a tool is missing or cluster is unreachable, so the orchestrator can distinguish "ran but failed checks" from "couldn't run"
- [ ] Every check is wrapped to capture its own success/failure without aborting the script
- [ ] GPU operator check inspects pod Ready conditions, not just pod phase
- [ ] CUDA smoke pod has the GPU pool toleration
- [ ] CUDA smoke waits for `phase=Succeeded` directly so short-lived `nvidia-smi` pods do not false-fail before becoming Ready
- [ ] CUDA smoke pod is deleted on every exit path (use trap or explicit cleanup at end of check)
- [ ] Blob probe runs inside the cluster using `BLOB_PROBE_SERVICE_ACCOUNT`, not the operator's local Azure login
- [ ] Report is written unconditionally on exit 0 and exit 1 (only exit 2 skips it)
- [ ] No persistent write/modify operations against cluster or Azure resources; temporary smoke/probe pods are cleaned up
- [ ] Per-check timeout — a hung Helm pod can't stall the whole validate
- [ ] `SKIP_CUDA` and `SKIP_BLOB_PROBE` flags work
- [ ] Summary line on stdout includes counts (consumed by `aks-bootstrap` in deploy mode)

## References

| Topic | Source |
|---|---|
| `az aks show` JSON shape | https://learn.microsoft.com/cli/azure/aks#az-aks-show |
| `kubectl get nodes -o json` | https://kubernetes.io/docs/reference/kubectl/jsonpath/ |
| GPU Operator readiness | https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html |