Aks Validate

AKS bootstrap validation

What this skill produces

A read-only bash script that runs ~10 checks against the live stack and emits:

• Stdout: per-check PASS, WARN, FAIL lines + summary.
• generated/stage07-<cluster-name>-validate-report.json — machine-readable report.
• Exit code: 0 if all checks PASS (WARN is non-fatal), 1 if any FAIL, 2 if preflight failed (missing tool, cluster unreachable).

This is the "cluster is ready" gate. Tolerant in orchestrator mode (see aks-bootstrap).

Inputs

Source from streaming-env.sh and any .env.* files written by upstream stages.

Required:

Optional (passed through from upstream stages for observed-vs-expected checks):

Output

generated/stage07-<cluster-name>-validate.sh and at runtime, generated/stage07-<cluster-name>-validate-report.json.

JSON shape:

{
  "cluster": "aks-nvcf-validation-westus3",
  "timestamp": "2026-05-27T12:34:56Z",
  "summary": { "pass": 8, "warn": 1, "fail": 0 },
  "checks": [
    { "id": "rg",          "status": "PASS", "detail": "RG exists in westus3" },
    { "id": "vnet",        "status": "PASS", "detail": "VNet 10.10.0.0/16; subnets: snet-aks-nodes, snet-aks-api-server" },
    { "id": "aks",         "status": "PASS", "detail": "Succeeded; v1.30.2" },
    { "id": "nodes",       "status": "PASS", "detail": "system=1, gpu=1 (Standard_NV36ads_A10_v5, zone 1)" },
    { "id": "acr-pull",    "status": "PASS", "detail": "AcrPull granted to kubelet identity" },
    { "id": "blob-rbac",   "status": "PASS", "detail": "Storage Blob Data Contributor on grantee" },
    { "id": "gpu-op",      "status": "PASS", "detail": "9/9 pods Ready; chart v25.3.0" },
    { "id": "gpu-resource","status": "PASS", "detail": "1 node advertises nvidia.com/gpu=1" },
    { "id": "cuda-smoke",  "status": "PASS", "detail": "nvidia-smi pod returned 0; A10 detected" },
    { "id": "blob-probe",  "status": "WARN", "detail": "Skipped (no STORAGE_ACCOUNT_NAME)" }
  ]
}

Checks

Each check has a per-check timeout (default 60s; cuda-smoke up to 5min).

Bash script (complete)

#!/bin/bash
set -uo pipefail   # NOT -e — we want to collect all check results, not bail on the first fail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
for envfile in .env.iam-vpc .env.aks-cluster .env.acr .env.blob-storage; do
  [[ -f "$REPO_ROOT/generated/$envfile" ]] && source "$REPO_ROOT/generated/$envfile"
done

: "${CLUSTER_NAME:?}" "${RESOURCE_GROUP:?}" "${AZURE_SUBSCRIPTION_ID:?}"
export KUBECONFIG="${KUBECONFIG:-/tmp/kubeconfig-${CLUSTER_NAME}}"
SKIP_CUDA="${SKIP_CUDA:-false}"
SKIP_BLOB_PROBE="${SKIP_BLOB_PROBE:-false}"
TIMEOUT_MIN="${TIMEOUT_MIN:-5}"

mkdir -p "$REPO_ROOT/generated"
REPORT="$REPO_ROOT/generated/stage07-${CLUSTER_NAME}-validate-report.json"
declare -a CHECKS
PASS=0; WARN=0; FAIL=0

record() {
  local id="$1" status="$2" detail="$3"
  CHECKS+=("$(printf '{"id":"%s","status":"%s","detail":"%s"}' "$id" "$status" "$(echo "$detail" | sed 's/"/\\"/g')")")
  case "$status" in PASS) ((PASS++));; WARN) ((WARN++));; FAIL) ((FAIL++));; esac
  printf "[%s] %s — %s\n" "$status" "$id" "$detail"
}

# Preflight
for tool in az kubectl jq; do
  command -v "$tool" >/dev/null || { echo "MISSING TOOL: $tool"; exit 2; }
done
az account show >/dev/null 2>&1 || { echo "az not authenticated"; exit 2; }
kubectl cluster-info >/dev/null 2>&1 || { echo "kubectl can't reach cluster $CLUSTER_NAME"; exit 2; }

# --- check: rg ---
LOC=$(az group show -n "$RESOURCE_GROUP" --query location -o tsv 2>/dev/null || echo "")
if [[ -n "$LOC" ]]; then record rg PASS "RG exists in $LOC"; else record rg FAIL "RG $RESOURCE_GROUP not found"; fi

# --- check: vnet ---
VNET_NAME="${VNET_NAME:-}"
NODE_SUBNET_NAME="${NODE_SUBNET_NAME:-snet-aks-nodes}"
API_SERVER_SUBNET_NAME="${API_SERVER_SUBNET_NAME:-snet-aks-api-server}"
if [[ -z "$VNET_NAME" ]]; then
  record vnet WARN "VNET_NAME unset; skipping VNet check"
else
  VNET_EXISTS=$(az network vnet show -n "$VNET_NAME" -g "$RESOURCE_GROUP" --query addressSpace.addressPrefixes[0] -o tsv 2>/dev/null || echo "")
  if [[ -z "$VNET_EXISTS" ]]; then
    record vnet FAIL "VNet $VNET_NAME not found in $RESOURCE_GROUP"
  else
    SUBNETS=$(az network vnet subnet list --vnet-name "$VNET_NAME" -g "$RESOURCE_GROUP" --query "[].name" -o tsv 2>/dev/null || echo "")
    MISSING=""
    [[ "$SUBNETS" != *"$NODE_SUBNET_NAME"* ]] && MISSING="$NODE_SUBNET_NAME"
    [[ "$SUBNETS" != *"$API_SERVER_SUBNET_NAME"* ]] && MISSING="${MISSING:+$MISSING, }$API_SERVER_SUBNET_NAME"
    if [[ -n "$MISSING" ]]; then
      record vnet FAIL "VNet $VNET_NAME exists ($VNET_EXISTS) but missing subnets: $MISSING"
    else
      record vnet PASS "VNet $VNET_NAME ($VNET_EXISTS); subnets: $NODE_SUBNET_NAME, $API_SERVER_SUBNET_NAME"
    fi
  fi
fi

# --- check: aks ---
STATE=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query provisioningState -o tsv 2>/dev/null || echo "")
OIDC=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query oidcIssuerProfile.issuerUrl -o tsv 2>/dev/null || echo "")
WI=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query securityProfile.workloadIdentity.enabled -o tsv 2>/dev/null || echo "")
K8S_VER=$(az aks show -n "$CLUSTER_NAME" -g "$RESOURCE_GROUP" --query currentKubernetesVersion -o tsv 2>/dev/null || echo "")
if [[ "$STATE" == "Succeeded" && -n "$OIDC" && "$WI" == "true" ]]; then
  record aks PASS "Succeeded; v${K8S_VER}; OIDC + workload identity enabled"
else
  record aks FAIL "state=$STATE oidc=${OIDC:-unset} wi=${WI:-unset}"
fi

# --- check: nodes ---
GPU_POOL_NAME="${GPU_POOL_NAME:-}"
GPU_VM_SIZE="${GPU_VM_SIZE:-}"
GPU_POOL_ZONE="${GPU_POOL_ZONE:-}"
POOL_JSON=$(az aks nodepool list -g "$RESOURCE_GROUP" --cluster-name "$CLUSTER_NAME" -o json 2>/dev/null || echo "[]")
SYSTEM_COUNT=$(echo "$POOL_JSON" | jq '[.[] | select(.mode=="System")] | length')
GPU_COUNT=$(echo "$POOL_JSON" | jq --arg n "${GPU_POOL_NAME:-gpu}" '[.[] | select(.name==$n or .mode=="User")] | length')
if [[ "$SYSTEM_COUNT" -eq 0 ]]; then
  record nodes FAIL "No system node pool found"
elif [[ "$GPU_COUNT" -eq 0 ]]; then
  record nodes FAIL "No user/GPU node pool found (expected name=${GPU_POOL_NAME:-gpu})"
else
  DETAIL="system=${SYSTEM_COUNT}, gpu=${GPU_COUNT}"
  NODE_STATUS="PASS"
  if [[ -n "$GPU_POOL_NAME" && -n "$GPU_VM_SIZE" ]]; then
    ACTUAL_SIZE=$(echo "$POOL_JSON" | jq -r --arg n "$GPU_POOL_NAME" '.[] | select(.name==$n) | .vmSize // ""')
    if [[ -n "$ACTUAL_SIZE" && "$ACTUAL_SIZE" != "$GPU_VM_SIZE" ]]; then
      DETAIL="$DETAIL; vm-size mismatch: expected=$GPU_VM_SIZE actual=$ACTUAL_SIZE"
      NODE_STATUS="FAIL"
    else
      DETAIL="$DETAIL; vm-size=$ACTUAL_SIZE"
    fi
  fi
  if [[ -n "$GPU_POOL_NAME" && -n "$GPU_POOL_ZONE" ]]; then
    ACTUAL_ZONES=$(echo "$POOL_JSON" | jq -r --arg n "$GPU_POOL_NAME" '.[] | select(.name==$n) | (.availabilityZones // []) | join(",")')
    DETAIL="$DETAIL; zone=${ACTUAL_ZONES:-unset}"
  fi
  record nodes "$NODE_STATUS" "$DETAIL"
fi

# --- check: acr-pull ---
if [[ -z "${ACR_NAME:-}" || -z "${KUBELET_IDENTITY_OBJECT_ID:-}" ]]; then
  record acr-pull WARN "ACR_NAME or KUBELET_IDENTITY_OBJECT_ID unset; skipping AcrPull check"
else
  ACR_ID=$(az acr show -n "$ACR_NAME" -g "$RESOURCE_GROUP" --query id -o tsv 2>/dev/null || echo "")
  if [[ -z "$ACR_ID" ]]; then
    record acr-pull FAIL "ACR $ACR_NAME not found in $RESOURCE_GROUP"
  else
    ROLE_COUNT=$(az role assignment list --scope "$ACR_ID" \
      --query "[?principalId=='${KUBELET_IDENTITY_OBJECT_ID}' && roleDefinitionName=='AcrPull'] | length(@)" \
      -o tsv 2>/dev/null || echo "0")
    if [[ "${ROLE_COUNT:-0}" -gt 0 ]]; then
      record acr-pull PASS "AcrPull granted to kubelet identity on $ACR_NAME"
    else
      record acr-pull FAIL "AcrPull not found on $ACR_NAME for kubelet identity $KUBELET_IDENTITY_OBJECT_ID"
    fi
  fi
fi

# --- check: blob-rbac ---
if [[ -z "${STORAGE_ACCOUNT_NAME:-}" ]]; then
  record blob-rbac WARN "STORAGE_ACCOUNT_NAME unset; skipping Blob RBAC check"
else
  SA_ID=$(az storage account show -n "$STORAGE_ACCOUNT_NAME" -g "$RESOURCE_GROUP" --query id -o tsv 2>/dev/null || echo "")
  if [[ -z "$SA_ID" ]]; then
    record blob-rbac FAIL "Storage account $STORAGE_ACCOUNT_NAME not found"
  else
    BLOB_ROLE_COUNT=$(az role assignment list --scope "$SA_ID" \
      --query "[?roleDefinitionName=='Storage Blob Data Contributor'] | length(@)" \
      -o tsv 2>/dev/null || echo "0")
    if [[ "${BLOB_ROLE_COUNT:-0}" -gt 0 ]]; then
      record blob-rbac PASS "Storage Blob Data Contributor assigned on $STORAGE_ACCOUNT_NAME"
    else
      record blob-rbac FAIL "Storage Blob Data Contributor not found on $STORAGE_ACCOUNT_NAME"
    fi
  fi
fi

# --- check: gpu-op ---
GPU_OP_NS="gpu-operator"
GPU_OP_TIMEOUT="${TIMEOUT_MIN}m"
PODS_JSON=$(kubectl get pods -n "$GPU_OP_NS" -o json 2>/dev/null || echo '{"items":[]}')
TOTAL=$(echo "$PODS_JSON" | jq '.items | length')
if [[ "$TOTAL" -eq 0 ]]; then
  record gpu-op FAIL "No pods found in namespace $GPU_OP_NS"
else
  NOT_READY=$(echo "$PODS_JSON" | jq '[.items[] | select(.status.phase != "Succeeded") | select(((.status.conditions // []) | map(select(.type=="Ready" and .status=="True")) | length) == 0)] | length')
  CHART_VER=$(kubectl get helmrelease -n "$GPU_OP_NS" gpu-operator -o jsonpath='{.spec.chart.spec.version}' 2>/dev/null || echo "")
  DETAIL="${TOTAL} pods; ready-condition-missing=${NOT_READY}${CHART_VER:+; chart=$CHART_VER}"
  if [[ "$NOT_READY" -eq 0 ]]; then
    record gpu-op PASS "$DETAIL"
  else
    record gpu-op FAIL "$DETAIL"
  fi
fi

# --- check: gpu-resource ---
GPU_NODE_COUNT=$(kubectl get nodes -o json 2>/dev/null \
  | jq '[.items[] | select((.status.capacity["nvidia.com/gpu"] // "0" | tonumber) >= 1)] | length')
if [[ "${GPU_NODE_COUNT:-0}" -gt 0 ]]; then
  record gpu-resource PASS "${GPU_NODE_COUNT} node(s) advertise nvidia.com/gpu >= 1"
else
  record gpu-resource FAIL "No nodes advertise nvidia.com/gpu (GPU operator may not be ready)"
fi

# --- check: cuda-smoke ---
if [[ "$SKIP_CUDA" == "true" ]]; then
  record cuda-smoke WARN "Skipped (SKIP_CUDA=true)"
else
  SMOKE_POD="validate-cuda-smoke-$$"
  CUDA_IMAGE="${CUDA_IMAGE:-nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04}"
  GPU_TAINT_KEY="${GPU_TAINT_KEY:-nvidia.com/gpu}"
  # Ensure pod is deleted on every exit path from this check
  cleanup_smoke() { kubectl delete pod "$SMOKE_POD" --ignore-not-found >/dev/null 2>&1 || true; }
  trap cleanup_smoke EXIT
  # Schedule onto a GPU node via the resource limit alone (no node-type selector —
  # the GPU pool is labeled nvidia.com/gpu=true, never node-type=gpu). The toleration
  # is harmless when the pool has no taint (repo default) and works if GPU_NODE_TAINTS is set.
  cat <<EOF | kubectl apply -f - >/dev/null 2>&1
apiVersion: v1
kind: Pod
metadata: { name: $SMOKE_POD, namespace: default }
spec:
  restartPolicy: Never
  tolerations:
    - { key: "$GPU_TAINT_KEY", operator: Exists, effect: NoSchedule }
  containers:
    - name: nvidia-smi
      image: $CUDA_IMAGE
      command: ["nvidia-smi"]
      resources: { limits: { nvidia.com/gpu: 1 } }
EOF
  if kubectl wait pod "$SMOKE_POD" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${TIMEOUT_MIN}m" >/dev/null 2>&1; then
    GPU_MODEL=$(kubectl logs "$SMOKE_POD" 2>/dev/null | grep -m1 "GPU" | sed 's/^[[:space:]]*//' || echo "unknown")
    record cuda-smoke PASS "nvidia-smi exited 0; ${GPU_MODEL}"
  else
    POD_PHASE=$(kubectl get pod "$SMOKE_POD" -o jsonpath='{.status.phase}' 2>/dev/null || echo "unknown")
    record cuda-smoke FAIL "nvidia-smi pod did not complete (phase=$POD_PHASE)"
  fi
  cleanup_smoke
  trap - EXIT
fi

# --- check: blob-probe ---
if [[ "$SKIP_BLOB_PROBE" == "true" || -z "${STORAGE_ACCOUNT_NAME:-}" || -z "${BLOB_PROBE_SERVICE_ACCOUNT:-}" ]]; then
  record blob-probe WARN "Skipped (SKIP_BLOB_PROBE=${SKIP_BLOB_PROBE} STORAGE_ACCOUNT_NAME=${STORAGE_ACCOUNT_NAME:-unset} BLOB_PROBE_SERVICE_ACCOUNT=${BLOB_PROBE_SERVICE_ACCOUNT:-unset})"
else
  BLOB_PROBE_NAMESPACE="${BLOB_PROBE_NAMESPACE:-default}"
  BLOB_PROBE_IMAGE="${BLOB_PROBE_IMAGE:-mcr.microsoft.com/azure-cli:2.61.0}"
  BLOB_CONTAINER="$(echo "${STORAGE_CONTAINERS:-scratch}" | cut -d, -f1)"
  BLOB_POD="validate-blob-probe-$$"
  cleanup_blob_probe() { kubectl delete pod "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true; }
  trap cleanup_blob_probe EXIT
  kubectl run "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" \
    --image="$BLOB_PROBE_IMAGE" \
    --restart=Never \
    --overrides="{\"spec\":{\"serviceAccountName\":\"${BLOB_PROBE_SERVICE_ACCOUNT}\"}}" \
    -- /bin/sh -lc "az storage blob list --account-name '${STORAGE_ACCOUNT_NAME}' --container-name '${BLOB_CONTAINER}' --auth-mode login --output none" >/dev/null 2>&1
  if kubectl wait pod "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${TIMEOUT_MIN}m" >/dev/null 2>&1; then
    record blob-probe PASS "workload identity pod listed blobs in $STORAGE_ACCOUNT_NAME/$BLOB_CONTAINER using serviceAccount=$BLOB_PROBE_SERVICE_ACCOUNT"
  else
    POD_PHASE=$(kubectl get pod "$BLOB_POD" -n "$BLOB_PROBE_NAMESPACE" -o jsonpath='{.status.phase}' 2>/dev/null || echo "unknown")
    record blob-probe FAIL "workload identity Blob probe failed (phase=$POD_PHASE serviceAccount=$BLOB_PROBE_SERVICE_ACCOUNT) — check federated credential / RBAC"
  fi
  cleanup_blob_probe
  trap - EXIT
fi

# --- emit report ---
printf '{ "cluster":"%s","timestamp":"%s","summary":{"pass":%d,"warn":%d,"fail":%d},"checks":[%s] }\n' \
  "$CLUSTER_NAME" "$(date -u +%FT%TZ)" "$PASS" "$WARN" "$FAIL" \
  "$(IFS=,; echo "${CHECKS[*]}")" | jq . > "$REPORT"

echo
echo "=== Summary: PASS=$PASS WARN=$WARN FAIL=$FAIL ==="
echo "Report: $REPORT"
[[ "$FAIL" -gt 0 ]] && exit 1 || exit 0

Tolerant execution

When run from aks-bootstrap deploy mode, the orchestrator reports the exit code but does NOT treat exit 1 as a bootstrap failure — stages 01–06 have already succeeded. The validate stage is diagnostic, not constructive. See aks-bootstrap for the contract.

Validation checklist (for the generator)

• Script uses set -uo pipefail (NOT -e — needs to keep collecting results past failures)
• Preflight exits with code 2 (not 1) if a tool is missing or cluster is unreachable, so the orchestrator can distinguish "ran but failed checks" from "couldn't run"
• Every check is wrapped to capture its own success/failure without aborting the script
• GPU operator check inspects pod Ready conditions, not just pod phase
• CUDA smoke pod has the GPU pool toleration
• CUDA smoke waits for phase=Succeeded directly so short-lived nvidia-smi pods do not false-fail before becoming Ready
• CUDA smoke pod is deleted on every exit path (use trap or explicit cleanup at end of check)
• Blob probe runs inside the cluster using BLOB_PROBE_SERVICE_ACCOUNT, not the operator's local Azure login
• Report is written unconditionally on exit 0 and exit 1 (only exit 2 skips it)
• No persistent write/modify operations against cluster or Azure resources; temporary smoke/probe pods are cleaned up
• Per-check timeout — a hung Helm pod can't stall the whole validate
• SKIP_CUDA and SKIP_BLOB_PROBE flags work
• Summary line on stdout includes counts (consumed by aks-bootstrap in deploy mode)

References