EKS Foundation Validation
What This Skill Produces
Generate generated/<cluster-name>-validate.sh, a validation script that emits
PASS/WARN/FAIL lines and writes generated/<cluster-name>-validate-report.json.
The report shape intentionally matches the Azure aks-validate style:
{
"cluster": "eks-nvcf-validation-usw2",
"timestamp": "2026-05-27T12:34:56Z",
"summary": { "pass": 8, "warn": 1, "fail": 0 },
"checks": [
{ "id": "account", "status": "PASS", "detail": "active account matches 123456789012 in us-west-2" },
{ "id": "vpc", "status": "PASS", "detail": "VPC vpc-abc 10.10.0.0/16; subnets: subnet-a, subnet-b, subnet-c" },
{ "id": "eks", "status": "PASS", "detail": "ACTIVE; v1.30; OIDC enabled" },
{ "id": "nodes", "status": "PASS", "detail": "system=1, compute=1, gpu=1 (g6.12xlarge, us-west-2a)" },
{ "id": "ecr-pull", "status": "PASS", "detail": "ECR read allowed for node role(s)" },
{ "id": "s3-rbac", "status": "WARN", "detail": "Skipped (no S3_BUCKET_NAME)" },
{ "id": "gpu-op", "status": "PASS", "detail": "9/9 pods Ready; chart v25.3.1" },
{ "id": "gpu-resource", "status": "PASS", "detail": "1 node advertises nvidia.com/gpu=4" },
{ "id": "cuda-smoke", "status": "PASS", "detail": "nvidia-smi pod returned 0; L40S detected" },
{ "id": "s3-probe", "status": "WARN", "detail": "Skipped (no S3_PROBE_SERVICE_ACCOUNT)" }
]
}
Inputs
Required:
Input
Notes
CLUSTER_NAME
EKS cluster name and kubeconfig context.
AWS_REGION
Target AWS region.
Optional:
Input
Default
Notes
AWS_ACCOUNT_ID
unset
If set, active caller account must match.
ECR_REPOSITORY_PREFIX
${CLUSTER_NAME}
Prefix used by ecr and ecr-mirror.
GPU_OPERATOR_VERSION
unset
If set, checked against Helm metadata when available.
CUDA_IMAGE
nvidia/cuda:12.4.0-base-ubuntu22.04
Image for smoke test.
SKIP_CUDA
false
Skip CUDA smoke with WARN.
ALLOW_GPU_ZERO
false
Treat no GPU nodes/resources as WARN instead of FAIL.
S3_BUCKET_NAME
unset
Enables optional S3 RBAC check.
S3_PROBE_NAMESPACE
default
Namespace for optional in-cluster S3 probe.
S3_PROBE_SERVICE_ACCOUNT
unset
ServiceAccount expected to have S3 access, usually via IRSA/EKS Pod Identity.
S3_PROBE_IMAGE
amazon/aws-cli:2.15.0
Image for optional in-cluster S3 probe.
SKIP_S3_PROBE
false
Skip S3 probe with WARN.
TIMEOUT_MIN
5
Per-check timeout cap.
Required tools:
Check Parity
Azure check
AWS equivalent
rg
account validates active AWS account and region context.
vnet
vpc validates EKS VPC, CIDR, and attached subnets.
aks
eks validates EKS ACTIVE, Kubernetes version, and OIDC issuer.
nodes
nodes validates system/compute/GPU node labels plus GPU instance type/zone detail.
acr-pull
ecr-pull validates ECR read access for EKS node role(s).
blob-rbac
s3-rbac optionally validates S3 permissions when S3_BUCKET_NAME is set.
gpu-op
gpu-op validates GPU Operator pod readiness and chart version when discoverable.
gpu-resource
gpu-resource validates advertised nvidia.com/gpu.
cuda-smoke
cuda-smoke runs an ephemeral nvidia-smi pod and cleans it up.
blob-probe
s3-probe optionally runs an in-cluster AWS CLI pod using the supplied service account.
Bash Script
Generate this script shape:
#!/bin/bash
set -uo pipefail
: "${CLUSTER_NAME:?Set CLUSTER_NAME before running}"
: "${AWS_REGION:?Set AWS_REGION before running}"
ECR_REPOSITORY_PREFIX="${ECR_REPOSITORY_PREFIX:-$CLUSTER_NAME}"
CUDA_IMAGE="${CUDA_IMAGE:-nvidia/cuda:12.4.0-base-ubuntu22.04}"
SKIP_CUDA="${SKIP_CUDA:-false}"
ALLOW_GPU_ZERO="${ALLOW_GPU_ZERO:-false}"
S3_PROBE_NAMESPACE="${S3_PROBE_NAMESPACE:-default}"
S3_PROBE_IMAGE="${S3_PROBE_IMAGE:-amazon/aws-cli:2.15.0}"
SKIP_S3_PROBE="${SKIP_S3_PROBE:-false}"
TIMEOUT_MIN="${TIMEOUT_MIN:-5}"
for tool in aws kubectl jq; do
command -v "$tool" >/dev/null 2>&1 || {
echo "ERROR: $tool is required but not installed" >&2
exit 2
}
done
kubectl --context "$CLUSTER_NAME" get nodes >/dev/null 2>&1 || {
echo "ERROR: cannot reach cluster context $CLUSTER_NAME" >&2
exit 2
}
mkdir -p generated
REPORT_FILE="generated/${CLUSTER_NAME}-validate-report.json"
PASS=0; WARN=0; FAIL=0
CHECKS=()
NODE_ROLE_ARNS=()
json_escape() {
jq -Rn --arg s "$1" '$s'
}
record() {
local id="$1" status="$2" detail="$3"
CHECKS+=("{\"id\":$(json_escape "$id"),\"status\":$(json_escape "$status"),\"detail\":$(json_escape "$detail")}")
case "$status" in
PASS) ((PASS++)) ;;
WARN) ((WARN++)) ;;
FAIL) ((FAIL++)) ;;
esac
printf '[%s] %s - %s\n' "$status" "$id" "$detail"
}
# --- check: account ---
ACTIVE_ACCOUNT="$(aws sts get-caller-identity --query Account --output text 2>/dev/null || true)"
if [[ -z "$ACTIVE_ACCOUNT" || "$ACTIVE_ACCOUNT" == "None" ]]; then
record account FAIL "cannot resolve active AWS account"
elif [[ -n "${AWS_ACCOUNT_ID:-}" && "$ACTIVE_ACCOUNT" != "$AWS_ACCOUNT_ID" ]]; then
record account FAIL "active account $ACTIVE_ACCOUNT does not match expected $AWS_ACCOUNT_ID"
elif [[ -n "${AWS_ACCOUNT_ID:-}" ]]; then
record account PASS "active account matches $AWS_ACCOUNT_ID in $AWS_REGION"
else
record account WARN "AWS_ACCOUNT_ID unset; active account is $ACTIVE_ACCOUNT in $AWS_REGION"
fi
CLUSTER_JSON="$(aws eks describe-cluster --name "$CLUSTER_NAME" --region "$AWS_REGION" --output json 2>/dev/null || true)"
CLUSTER_READABLE=true
if [[ -z "$CLUSTER_JSON" ]]; then
record eks FAIL "cluster $CLUSTER_NAME not found or not readable in $AWS_REGION"
CLUSTER_JSON='{"cluster":{}}'
CLUSTER_READABLE=false
fi
# --- check: vpc ---
VPC_ID="$(echo "$CLUSTER_JSON" | jq -r '.cluster.resourcesVpcConfig.vpcId // ""')"
if [[ -z "$VPC_ID" ]]; then
record vpc FAIL "cluster VPC ID unavailable"
else
VPC_CIDR="$(aws ec2 describe-vpcs --region "$AWS_REGION" --vpc-ids "$VPC_ID" --query 'Vpcs[0].CidrBlock' --output text 2>/dev/null || true)"
mapfile -t SUBNET_IDS < <(echo "$CLUSTER_JSON" | jq -r '.cluster.resourcesVpcConfig.subnetIds[]?')
if [[ "${#SUBNET_IDS[@]}" -eq 0 ]]; then
record vpc FAIL "VPC $VPC_ID has no EKS subnet IDs in cluster config"
else
SUBNET_NAMES="$(aws ec2 describe-subnets --region "$AWS_REGION" --subnet-ids "${SUBNET_IDS[@]}" \
--query 'Subnets[].{id:SubnetId,name:Tags[?Key==`Name`]|[0].Value,cidr:CidrBlock}' \
--output json 2>/dev/null | jq -r '.[] | (.name // .id) + " " + .cidr' | paste -sd ', ' -)"
record vpc PASS "VPC $VPC_ID ${VPC_CIDR:-unknown}; subnets: ${SUBNET_NAMES:-unknown}"
fi
fi
# --- check: eks ---
# Skip when describe-cluster was unreadable; the empty-CLUSTER_JSON guard above
# already recorded the eks FAIL, so re-recording here would double-count it.
if [[ "$CLUSTER_READABLE" == "true" ]]; then
EKS_STATUS="$(echo "$CLUSTER_JSON" | jq -r '.cluster.status // ""')"
EKS_VERSION="$(echo "$CLUSTER_JSON" | jq -r '.cluster.version // ""')"
OIDC_ISSUER="$(echo "$CLUSTER_JSON" | jq -r '.cluster.identity.oidc.issuer // ""')"
if [[ "$EKS_STATUS" == "ACTIVE" && -n "$OIDC_ISSUER" ]]; then
record eks PASS "ACTIVE; v${EKS_VERSION:-unknown}; OIDC enabled"
elif [[ "$EKS_STATUS" == "ACTIVE" ]]; then
record eks FAIL "ACTIVE but OIDC issuer is unset"
else
record eks FAIL "status=${EKS_STATUS:-UNKNOWN}; version=${EKS_VERSION:-unknown}; oidc=${OIDC_ISSUER:-unset}"
fi
fi
# --- collect node group roles for ECR/S3 checks ---
mapfile -t NODEGROUPS < <(aws eks list-nodegroups --cluster-name "$CLUSTER_NAME" --region "$AWS_REGION" --query 'nodegroups[]' --output text 2>/dev/null | tr '\t' '\n')
for ng in "${NODEGROUPS[@]}"; do
[[ -z "$ng" ]] && continue
role_arn="$(aws eks describe-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$ng" --region "$AWS_REGION" --query 'nodegroup.nodeRole' --output text 2>/dev/null || true)"
[[ -n "$role_arn" && "$role_arn" != "None" ]] && NODE_ROLE_ARNS+=("$role_arn")
done
# --- check: nodes ---
NODES_JSON="$(kubectl --context "$CLUSTER_NAME" get nodes -o json 2>/dev/null || echo '{"items":[]}')"
SYSTEM_COUNT="$(echo "$NODES_JSON" | jq '[.items[] | select(.metadata.labels["node-type"]=="system")] | length')"
COMPUTE_COUNT="$(echo "$NODES_JSON" | jq '[.items[] | select(.metadata.labels["node-type"]=="compute")] | length')"
GPU_COUNT="$(echo "$NODES_JSON" | jq '[.items[] | select(.metadata.labels["node-type"]=="gpu")] | length')"
NOT_READY="$(echo "$NODES_JSON" | jq '[.items[] | select((.status.conditions // []) | map(select(.type=="Ready" and .status=="True")) | length == 0)] | length')"
GPU_DETAILS="$(echo "$NODES_JSON" | jq -r '[.items[] | select(.metadata.labels["node-type"]=="gpu") | ((.metadata.labels["node.kubernetes.io/instance-type"] // "unknown") + ", " + (.metadata.labels["topology.kubernetes.io/zone"] // "unknown"))] | unique | join("; ")')"
NODE_DETAIL="system=${SYSTEM_COUNT}, compute=${COMPUTE_COUNT}, gpu=${GPU_COUNT}${GPU_DETAILS:+ (${GPU_DETAILS})}"
if [[ "$NOT_READY" -gt 0 ]]; then
record nodes FAIL "$NODE_DETAIL; not-ready=${NOT_READY}"
elif [[ "$SYSTEM_COUNT" -eq 0 ]]; then
record nodes FAIL "$NODE_DETAIL; missing node-type=system"
elif [[ "$GPU_COUNT" -eq 0 && "$ALLOW_GPU_ZERO" == "true" ]]; then
record nodes WARN "$NODE_DETAIL; no GPU nodes and ALLOW_GPU_ZERO=true"
elif [[ "$GPU_COUNT" -eq 0 ]]; then
record nodes FAIL "$NODE_DETAIL; missing node-type=gpu"
else
record nodes PASS "$NODE_DETAIL"
fi
# --- check: ecr-pull ---
ECR_REPO_JSON="$(aws ecr describe-repositories --region "$AWS_REGION" \
--query "repositories[?starts_with(repositoryName, '${ECR_REPOSITORY_PREFIX}/')].{name:repositoryName,arn:repositoryArn}" \
--output json 2>/dev/null || echo '[]')"
ECR_REPOS="$(echo "$ECR_REPO_JSON" | jq -r '.[].name' | paste -sd ' ' -)"
mapfile -t ECR_REPO_ARNS < <(echo "$ECR_REPO_JSON" | jq -r '.[].arn')
ECR_REPO_COUNT="${#ECR_REPO_ARNS[@]}"
if [[ "${#NODE_ROLE_ARNS[@]}" -eq 0 ]]; then
record ecr-pull WARN "no EKS node role ARNs found; cannot prove ECR pull access"
else
ECR_AUTH_ACTION=(ecr:GetAuthorizationToken)
ECR_REPO_ACTIONS=(ecr:BatchCheckLayerAvailability ecr:GetDownloadUrlForLayer ecr:BatchGetImage)
all_allowed=true
sim_available=true
for role_arn in "${NODE_ROLE_ARNS[@]}"; do
auth_sim_json="$(aws iam simulate-principal-policy \
--policy-source-arn "$role_arn" \
--action-names "${ECR_AUTH_ACTION[@]}" \
--resource-arns "*" \
--output json 2>/dev/null || true)"
repo_sim_json=""
if [[ "${#ECR_REPO_ARNS[@]}" -gt 0 ]]; then
repo_sim_json="$(aws iam simulate-principal-policy \
--policy-source-arn "$role_arn" \
--action-names "${ECR_REPO_ACTIONS[@]}" \
--resource-arns "${ECR_REPO_ARNS[@]}" \
--output json 2>/dev/null || true)"
fi
if [[ -z "$auth_sim_json" || ( "${#ECR_REPO_ARNS[@]}" -gt 0 && -z "$repo_sim_json" ) ]]; then
sim_available=false
role_name="${role_arn##*/}"
attached="$(aws iam list-attached-role-policies --role-name "$role_name" --query 'AttachedPolicies[].PolicyArn' --output text 2>/dev/null || true)"
if [[ "$attached" != *"AmazonEC2ContainerRegistryReadOnly"* && "$attached" != *"AmazonEC2ContainerRegistryPowerUser"* ]]; then
all_allowed=false
fi
else
denied="$(echo "$auth_sim_json" | jq '[.EvaluationResults[] | select(.EvalDecision != "allowed")] | length')"
if [[ "${#ECR_REPO_ARNS[@]}" -gt 0 ]]; then
repo_denied="$(echo "$repo_sim_json" | jq '[.EvaluationResults[] | select(.EvalDecision != "allowed")] | length')"
denied=$((denied + repo_denied))
fi
[[ "$denied" -gt 0 ]] && all_allowed=false
fi
done
if [[ "$all_allowed" == "true" ]]; then
if [[ "$ECR_REPO_COUNT" -gt 0 ]]; then
record ecr-pull PASS "ECR read allowed for node role(s); ${ECR_REPO_COUNT} repo(s) under ${ECR_REPOSITORY_PREFIX}/"
else
record ecr-pull PASS "ECR read allowed for node role(s); no repos yet under ${ECR_REPOSITORY_PREFIX}/"
fi
elif [[ "$sim_available" == "false" ]]; then
record ecr-pull WARN "could not prove ECR read for every node role; iam:SimulatePrincipalPolicy unavailable and managed policy fallback did not match"
else
record ecr-pull FAIL "one or more node roles lack required ECR read actions"
fi
fi
# --- check: s3-rbac ---
if [[ -z "${S3_BUCKET_NAME:-}" ]]; then
record s3-rbac WARN "Skipped (no S3_BUCKET_NAME)"
elif [[ "${#NODE_ROLE_ARNS[@]}" -eq 0 && -z "${S3_PROBE_ROLE_ARN:-}" ]]; then
record s3-rbac WARN "S3_BUCKET_NAME set, but no role ARN is available for IAM simulation"
else
S3_ROLE_ARN="${S3_PROBE_ROLE_ARN:-${NODE_ROLE_ARNS[0]}}"
s3_sim="$(aws iam simulate-principal-policy \
--policy-source-arn "$S3_ROLE_ARN" \
--action-names s3:ListBucket s3:GetObject \
--resource-arns "arn:aws:s3:::${S3_BUCKET_NAME}" "arn:aws:s3:::${S3_BUCKET_NAME}/*" \
--output json 2>/dev/null || true)"
if [[ -z "$s3_sim" ]]; then
record s3-rbac WARN "Could not simulate S3 access for $S3_ROLE_ARN"
else
denied="$(echo "$s3_sim" | jq '[.EvaluationResults[] | select(.EvalDecision != "allowed")] | length')"
if [[ "$denied" -eq 0 ]]; then
record s3-rbac PASS "S3 List/Get allowed on $S3_BUCKET_NAME for ${S3_ROLE_ARN##*/}"
else
record s3-rbac FAIL "S3 List/Get not fully allowed on $S3_BUCKET_NAME for ${S3_ROLE_ARN##*/}"
fi
fi
fi
# --- check: gpu-op ---
PODS_JSON="$(kubectl --context "$CLUSTER_NAME" get pods -n gpu-operator -o json 2>/dev/null || echo '{"items":[]}')"
GPU_OP_TOTAL="$(echo "$PODS_JSON" | jq '.items | length')"
GPU_OP_NOT_READY="$(echo "$PODS_JSON" | jq '[.items[] | select(.status.phase != "Succeeded") | select(((.status.conditions // []) | map(select(.type=="Ready" and .status=="True")) | length) == 0)] | length')"
GPU_OP_READY=$((GPU_OP_TOTAL - GPU_OP_NOT_READY))
CHART_VER="$(helm --kube-context "$CLUSTER_NAME" list -n gpu-operator -o json 2>/dev/null | jq -r '.[] | select(.name=="gpu-operator") | .chart // ""' | sed 's/^gpu-operator-v\?//' || true)"
if [[ "$GPU_OP_TOTAL" -eq 0 ]]; then
record gpu-op FAIL "no pods found in gpu-operator namespace"
elif [[ "$GPU_OP_NOT_READY" -eq 0 ]]; then
if [[ -n "${GPU_OPERATOR_VERSION:-}" && -n "$CHART_VER" && "$CHART_VER" != "${GPU_OPERATOR_VERSION#v}" ]]; then
record gpu-op WARN "${GPU_OP_READY}/${GPU_OP_TOTAL} pods Ready; chart v${CHART_VER}, expected v${GPU_OPERATOR_VERSION#v}"
else
record gpu-op PASS "${GPU_OP_READY}/${GPU_OP_TOTAL} pods Ready${CHART_VER:+; chart v$CHART_VER}"
fi
else
record gpu-op FAIL "${GPU_OP_READY}/${GPU_OP_TOTAL} pods Ready"
fi
# --- check: gpu-resource ---
GPU_RESOURCE_NODES="$(echo "$NODES_JSON" | jq '[.items[] | select((.status.allocatable["nvidia.com/gpu"] // "0" | tonumber) >= 1)] | length')"
GPU_RESOURCE_TOTAL="$(echo "$NODES_JSON" | jq '[.items[] | (.status.allocatable["nvidia.com/gpu"] // "0" | tonumber)] | add // 0')"
if [[ "$GPU_RESOURCE_NODES" -gt 0 ]]; then
record gpu-resource PASS "${GPU_RESOURCE_NODES} node(s) advertise nvidia.com/gpu=${GPU_RESOURCE_TOTAL}"
elif [[ "$ALLOW_GPU_ZERO" == "true" ]]; then
record gpu-resource WARN "no nodes advertise nvidia.com/gpu and ALLOW_GPU_ZERO=true"
else
record gpu-resource FAIL "no nodes advertise nvidia.com/gpu"
fi
# --- check: cuda-smoke ---
SMOKE_POD="aws-validate-cuda-smoke-$$"
cleanup_smoke() {
kubectl --context "$CLUSTER_NAME" delete pod "$SMOKE_POD" --ignore-not-found >/dev/null 2>&1 || true
}
if [[ "$SKIP_CUDA" == "true" ]]; then
record cuda-smoke WARN "Skipped (SKIP_CUDA=true)"
elif [[ "$GPU_RESOURCE_NODES" -eq 0 ]]; then
record cuda-smoke WARN "Skipped (no GPU nodes available)"
else
trap cleanup_smoke EXIT
kubectl --context "$CLUSTER_NAME" run "$SMOKE_POD" \
--image="$CUDA_IMAGE" \
--restart=Never \
--overrides='{"spec":{"nodeSelector":{"node-type":"gpu"},"tolerations":[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}],"containers":[{"name":"smoke","image":"'"$CUDA_IMAGE"'","command":["nvidia-smi"],"resources":{"limits":{"nvidia.com/gpu":"1"}}}]}}' \
>/dev/null 2>&1
end=$((SECONDS + TIMEOUT_MIN * 60))
phase=""
while (( SECONDS < end )); do
phase="$(kubectl --context "$CLUSTER_NAME" get pod "$SMOKE_POD" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
[[ "$phase" == "Succeeded" || "$phase" == "Failed" ]] && break
sleep 5
done
if [[ "$phase" == "Succeeded" ]]; then
GPU_MODEL="$(kubectl --context "$CLUSTER_NAME" logs "$SMOKE_POD" 2>/dev/null | grep -Eom1 'A10|L40S|T4|H100|A100|V100|L4' || true)"
record cuda-smoke PASS "nvidia-smi pod returned 0${GPU_MODEL:+; $GPU_MODEL detected}"
else
detail="phase=${phase:-UNKNOWN}"
pod_log="$(kubectl --context "$CLUSTER_NAME" logs "$SMOKE_POD" 2>/dev/null | tail -20 | tr '\n' ' ' || true)"
[[ -n "$pod_log" ]] && detail="$detail; logs: $pod_log"
record cuda-smoke FAIL "nvidia-smi pod did not complete successfully ($detail)"
fi
cleanup_smoke
trap - EXIT
fi
# --- check: s3-probe ---
S3_POD="aws-validate-s3-probe-$$"
cleanup_s3_probe() {
kubectl --context "$CLUSTER_NAME" delete pod "$S3_POD" -n "$S3_PROBE_NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true
}
if [[ "$SKIP_S3_PROBE" == "true" || -z "${S3_BUCKET_NAME:-}" || -z "${S3_PROBE_SERVICE_ACCOUNT:-}" ]]; then
record s3-probe WARN "Skipped (SKIP_S3_PROBE=${SKIP_S3_PROBE} S3_BUCKET_NAME=${S3_BUCKET_NAME:-unset} S3_PROBE_SERVICE_ACCOUNT=${S3_PROBE_SERVICE_ACCOUNT:-unset})"
else
trap cleanup_s3_probe EXIT
kubectl --context "$CLUSTER_NAME" run "$S3_POD" -n "$S3_PROBE_NAMESPACE" \
--image="$S3_PROBE_IMAGE" \
--restart=Never \
--overrides='{"spec":{"serviceAccountName":"'"$S3_PROBE_SERVICE_ACCOUNT"'","containers":[{"name":"aws","image":"'"$S3_PROBE_IMAGE"'","command":["aws","s3api","list-objects-v2","--bucket","'"$S3_BUCKET_NAME"'","--max-items","1"]}]}}' \
>/dev/null 2>&1
if kubectl --context "$CLUSTER_NAME" wait pod "$S3_POD" -n "$S3_PROBE_NAMESPACE" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${TIMEOUT_MIN}m" >/dev/null 2>&1; then
record s3-probe PASS "serviceAccount=$S3_PROBE_SERVICE_ACCOUNT listed $S3_BUCKET_NAME"
else
phase="$(kubectl --context "$CLUSTER_NAME" get pod "$S3_POD" -n "$S3_PROBE_NAMESPACE" -o jsonpath='{.status.phase}' 2>/dev/null || echo unknown)"
record s3-probe FAIL "S3 probe failed (phase=$phase serviceAccount=$S3_PROBE_SERVICE_ACCOUNT bucket=$S3_BUCKET_NAME)"
fi
cleanup_s3_probe
trap - EXIT
fi
# --- emit report ---
printf '{ "cluster": "%s", "timestamp": "%s", "summary": { "pass": %d, "warn": %d, "fail": %d }, "checks": [%s] }\n' \
"$CLUSTER_NAME" "$(date -u +%FT%TZ)" "$PASS" "$WARN" "$FAIL" "$(IFS=,; echo "${CHECKS[*]}")" \
| jq . > "$REPORT_FILE"
echo ""
echo "=== Summary: PASS=$PASS WARN=$WARN FAIL=$FAIL ==="
echo "Report: $REPORT_FILE"
[[ "$FAIL" -gt 0 ]] && exit 1 || exit 0
Validation Checklist
Report uses Azure-compatible cluster, timestamp, summary, checks[].id, checks[].status, and checks[].detail fields.
Preflight exits with code 2 if a tool is missing or the cluster is unreachable.
Account check validates AWS_ACCOUNT_ID when supplied.
VPC check reports the EKS VPC CIDR and attached subnet names/CIDRs.
EKS check requires ACTIVE and OIDC issuer.
Node check verifies node-type=system and node-type=gpu; ALLOW_GPU_ZERO=true downgrades missing GPU nodes to WARN.
ECR pull check proves required read actions with IAM simulation when possible, with managed-policy fallback.
Optional S3 RBAC and S3 probe checks WARN when not configured.
GPU Operator check inspects pod Ready conditions, not just pod phase.
CUDA smoke pod requests one GPU, waits for Succeeded, and is deleted on every exit path.
No secret values, kubeconfig contents, ECR passwords, or AWS credentials are printed.
1 --- 2 name: eks-validate 3 description: Generate a validation script for an AWS KAS/NVCF EKS foundation. Checks AWS account identity, VPC/subnets, EKS state and OIDC, node group composition, ECR pull permissions, optional S3 RBAC/probe, GPU Operator readiness, GPU resources, and an ephemeral CUDA smoke pod. 4 --- 5 6 <!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> 7 <!-- SPDX-License-Identifier: CC-BY-4.0 AND Apache-2.0 --> 8 9 # EKS Foundation Validation 10 11 ## What This Skill Produces 12 13 Generate `generated/<cluster-name>-validate.sh`, a validation script that emits 14 PASS/WARN/FAIL lines and writes `generated/<cluster-name>-validate-report.json`. 15 16 The report shape intentionally matches the Azure `aks-validate` style: 17 18 ```json 19 { 20 "cluster": "eks-nvcf-validation-usw2", 21 "timestamp": "2026-05-27T12:34:56Z", 22 "summary": { "pass": 8, "warn": 1, "fail": 0 }, 23 "checks": [ 24 { "id": "account", "status": "PASS", "detail": "active account matches 123456789012 in us-west-2" }, 25 { "id": "vpc", "status": "PASS", "detail": "VPC vpc-abc 10.10.0.0/16; subnets: subnet-a, subnet-b, subnet-c" }, 26 { "id": "eks", "status": "PASS", "detail": "ACTIVE; v1.30; OIDC enabled" }, 27 { "id": "nodes", "status": "PASS", "detail": "system=1, compute=1, gpu=1 (g6.12xlarge, us-west-2a)" }, 28 { "id": "ecr-pull", "status": "PASS", "detail": "ECR read allowed for node role(s)" }, 29 { "id": "s3-rbac", "status": "WARN", "detail": "Skipped (no S3_BUCKET_NAME)" }, 30 { "id": "gpu-op", "status": "PASS", "detail": "9/9 pods Ready; chart v25.3.1" }, 31 { "id": "gpu-resource", "status": "PASS", "detail": "1 node advertises nvidia.com/gpu=4" }, 32 { "id": "cuda-smoke", "status": "PASS", "detail": "nvidia-smi pod returned 0; L40S detected" }, 33 { "id": "s3-probe", "status": "WARN", "detail": "Skipped (no S3_PROBE_SERVICE_ACCOUNT)" } 34 ] 35 } 36 ``` 37 38 ## Inputs 39 40 Required: 41 42 | Input | Notes | 43 |---|---| 44 | `CLUSTER_NAME` | EKS cluster name and kubeconfig context. | 45 | `AWS_REGION` | Target AWS region. | 46 47 Optional: 48 49 | Input | Default | Notes | 50 |---|---|---| 51 | `AWS_ACCOUNT_ID` | unset | If set, active caller account must match. | 52 | `ECR_REPOSITORY_PREFIX` | `${CLUSTER_NAME}` | Prefix used by `ecr` and `ecr-mirror`. | 53 | `GPU_OPERATOR_VERSION` | unset | If set, checked against Helm metadata when available. | 54 | `CUDA_IMAGE` | `nvidia/cuda:12.4.0-base-ubuntu22.04` | Image for smoke test. | 55 | `SKIP_CUDA` | `false` | Skip CUDA smoke with WARN. | 56 | `ALLOW_GPU_ZERO` | `false` | Treat no GPU nodes/resources as WARN instead of FAIL. | 57 | `S3_BUCKET_NAME` | unset | Enables optional S3 RBAC check. | 58 | `S3_PROBE_NAMESPACE` | `default` | Namespace for optional in-cluster S3 probe. | 59 | `S3_PROBE_SERVICE_ACCOUNT` | unset | ServiceAccount expected to have S3 access, usually via IRSA/EKS Pod Identity. | 60 | `S3_PROBE_IMAGE` | `amazon/aws-cli:2.15.0` | Image for optional in-cluster S3 probe. | 61 | `SKIP_S3_PROBE` | `false` | Skip S3 probe with WARN. | 62 | `TIMEOUT_MIN` | `5` | Per-check timeout cap. | 63 64 Required tools: 65 66 - `aws` 67 - `kubectl` 68 - `jq` 69 70 ## Check Parity 71 72 | Azure check | AWS equivalent | 73 |---|---| 74 | `rg` | `account` validates active AWS account and region context. | 75 | `vnet` | `vpc` validates EKS VPC, CIDR, and attached subnets. | 76 | `aks` | `eks` validates EKS `ACTIVE`, Kubernetes version, and OIDC issuer. | 77 | `nodes` | `nodes` validates system/compute/GPU node labels plus GPU instance type/zone detail. | 78 | `acr-pull` | `ecr-pull` validates ECR read access for EKS node role(s). | 79 | `blob-rbac` | `s3-rbac` optionally validates S3 permissions when `S3_BUCKET_NAME` is set. | 80 | `gpu-op` | `gpu-op` validates GPU Operator pod readiness and chart version when discoverable. | 81 | `gpu-resource` | `gpu-resource` validates advertised `nvidia.com/gpu`. | 82 | `cuda-smoke` | `cuda-smoke` runs an ephemeral `nvidia-smi` pod and cleans it up. | 83 | `blob-probe` | `s3-probe` optionally runs an in-cluster AWS CLI pod using the supplied service account. | 84 85 ## Bash Script 86 87 Generate this script shape: 88 89 ```bash 90 #!/bin/bash 91 set -uo pipefail 92 93 : "${CLUSTER_NAME:?Set CLUSTER_NAME before running}" 94 : "${AWS_REGION:?Set AWS_REGION before running}" 95 96 ECR_REPOSITORY_PREFIX="${ECR_REPOSITORY_PREFIX:-$CLUSTER_NAME}" 97 CUDA_IMAGE="${CUDA_IMAGE:-nvidia/cuda:12.4.0-base-ubuntu22.04}" 98 SKIP_CUDA="${SKIP_CUDA:-false}" 99 ALLOW_GPU_ZERO="${ALLOW_GPU_ZERO:-false}" 100 S3_PROBE_NAMESPACE="${S3_PROBE_NAMESPACE:-default}" 101 S3_PROBE_IMAGE="${S3_PROBE_IMAGE:-amazon/aws-cli:2.15.0}" 102 SKIP_S3_PROBE="${SKIP_S3_PROBE:-false}" 103 TIMEOUT_MIN="${TIMEOUT_MIN:-5}" 104 105 for tool in aws kubectl jq; do 106 command -v "$tool" >/dev/null 2>&1 || { 107 echo "ERROR: $tool is required but not installed" >&2 108 exit 2 109 } 110 done 111 112 kubectl --context "$CLUSTER_NAME" get nodes >/dev/null 2>&1 || { 113 echo "ERROR: cannot reach cluster context $CLUSTER_NAME" >&2 114 exit 2 115 } 116 117 mkdir -p generated 118 REPORT_FILE="generated/${CLUSTER_NAME}-validate-report.json" 119 120 PASS=0; WARN=0; FAIL=0 121 CHECKS=() 122 NODE_ROLE_ARNS=() 123 124 json_escape() { 125 jq -Rn --arg s "$1" '$s' 126 } 127 128 record() { 129 local id="$1" status="$2" detail="$3" 130 CHECKS+=("{\"id\":$(json_escape "$id"),\"status\":$(json_escape "$status"),\"detail\":$(json_escape "$detail")}") 131 case "$status" in 132 PASS) ((PASS++)) ;; 133 WARN) ((WARN++)) ;; 134 FAIL) ((FAIL++)) ;; 135 esac 136 printf '[%s] %s - %s\n' "$status" "$id" "$detail" 137 } 138 139 # --- check: account --- 140 ACTIVE_ACCOUNT="$(aws sts get-caller-identity --query Account --output text 2>/dev/null || true)" 141 if [[ -z "$ACTIVE_ACCOUNT" || "$ACTIVE_ACCOUNT" == "None" ]]; then 142 record account FAIL "cannot resolve active AWS account" 143 elif [[ -n "${AWS_ACCOUNT_ID:-}" && "$ACTIVE_ACCOUNT" != "$AWS_ACCOUNT_ID" ]]; then 144 record account FAIL "active account $ACTIVE_ACCOUNT does not match expected $AWS_ACCOUNT_ID" 145 elif [[ -n "${AWS_ACCOUNT_ID:-}" ]]; then 146 record account PASS "active account matches $AWS_ACCOUNT_ID in $AWS_REGION" 147 else 148 record account WARN "AWS_ACCOUNT_ID unset; active account is $ACTIVE_ACCOUNT in $AWS_REGION" 149 fi 150 151 CLUSTER_JSON="$(aws eks describe-cluster --name "$CLUSTER_NAME" --region "$AWS_REGION" --output json 2>/dev/null || true)" 152 CLUSTER_READABLE=true 153 if [[ -z "$CLUSTER_JSON" ]]; then 154 record eks FAIL "cluster $CLUSTER_NAME not found or not readable in $AWS_REGION" 155 CLUSTER_JSON='{"cluster":{}}' 156 CLUSTER_READABLE=false 157 fi 158 159 # --- check: vpc --- 160 VPC_ID="$(echo "$CLUSTER_JSON" | jq -r '.cluster.resourcesVpcConfig.vpcId // ""')" 161 if [[ -z "$VPC_ID" ]]; then 162 record vpc FAIL "cluster VPC ID unavailable" 163 else 164 VPC_CIDR="$(aws ec2 describe-vpcs --region "$AWS_REGION" --vpc-ids "$VPC_ID" --query 'Vpcs[0].CidrBlock' --output text 2>/dev/null || true)" 165 mapfile -t SUBNET_IDS < <(echo "$CLUSTER_JSON" | jq -r '.cluster.resourcesVpcConfig.subnetIds[]?') 166 if [[ "${#SUBNET_IDS[@]}" -eq 0 ]]; then 167 record vpc FAIL "VPC $VPC_ID has no EKS subnet IDs in cluster config" 168 else 169 SUBNET_NAMES="$(aws ec2 describe-subnets --region "$AWS_REGION" --subnet-ids "${SUBNET_IDS[@]}" \ 170 --query 'Subnets[].{id:SubnetId,name:Tags[?Key==`Name`]|[0].Value,cidr:CidrBlock}' \ 171 --output json 2>/dev/null | jq -r '.[] | (.name // .id) + " " + .cidr' | paste -sd ', ' -)" 172 record vpc PASS "VPC $VPC_ID ${VPC_CIDR:-unknown}; subnets: ${SUBNET_NAMES:-unknown}" 173 fi 174 fi 175 176 # --- check: eks --- 177 # Skip when describe-cluster was unreadable; the empty-CLUSTER_JSON guard above 178 # already recorded the eks FAIL, so re-recording here would double-count it. 179 if [[ "$CLUSTER_READABLE" == "true" ]]; then 180 EKS_STATUS="$(echo "$CLUSTER_JSON" | jq -r '.cluster.status // ""')" 181 EKS_VERSION="$(echo "$CLUSTER_JSON" | jq -r '.cluster.version // ""')" 182 OIDC_ISSUER="$(echo "$CLUSTER_JSON" | jq -r '.cluster.identity.oidc.issuer // ""')" 183 if [[ "$EKS_STATUS" == "ACTIVE" && -n "$OIDC_ISSUER" ]]; then 184 record eks PASS "ACTIVE; v${EKS_VERSION:-unknown}; OIDC enabled" 185 elif [[ "$EKS_STATUS" == "ACTIVE" ]]; then 186 record eks FAIL "ACTIVE but OIDC issuer is unset" 187 else 188 record eks FAIL "status=${EKS_STATUS:-UNKNOWN}; version=${EKS_VERSION:-unknown}; oidc=${OIDC_ISSUER:-unset}" 189 fi 190 fi 191 192 # --- collect node group roles for ECR/S3 checks --- 193 mapfile -t NODEGROUPS < <(aws eks list-nodegroups --cluster-name "$CLUSTER_NAME" --region "$AWS_REGION" --query 'nodegroups[]' --output text 2>/dev/null | tr '\t' '\n') 194 for ng in "${NODEGROUPS[@]}"; do 195 [[ -z "$ng" ]] && continue 196 role_arn="$(aws eks describe-nodegroup --cluster-name "$CLUSTER_NAME" --nodegroup-name "$ng" --region "$AWS_REGION" --query 'nodegroup.nodeRole' --output text 2>/dev/null || true)" 197 [[ -n "$role_arn" && "$role_arn" != "None" ]] && NODE_ROLE_ARNS+=("$role_arn") 198 done 199 200 # --- check: nodes --- 201 NODES_JSON="$(kubectl --context "$CLUSTER_NAME" get nodes -o json 2>/dev/null || echo '{"items":[]}')" 202 SYSTEM_COUNT="$(echo "$NODES_JSON" | jq '[.items[] | select(.metadata.labels["node-type"]=="system")] | length')" 203 COMPUTE_COUNT="$(echo "$NODES_JSON" | jq '[.items[] | select(.metadata.labels["node-type"]=="compute")] | length')" 204 GPU_COUNT="$(echo "$NODES_JSON" | jq '[.items[] | select(.metadata.labels["node-type"]=="gpu")] | length')" 205 NOT_READY="$(echo "$NODES_JSON" | jq '[.items[] | select((.status.conditions // []) | map(select(.type=="Ready" and .status=="True")) | length == 0)] | length')" 206 GPU_DETAILS="$(echo "$NODES_JSON" | jq -r '[.items[] | select(.metadata.labels["node-type"]=="gpu") | ((.metadata.labels["node.kubernetes.io/instance-type"] // "unknown") + ", " + (.metadata.labels["topology.kubernetes.io/zone"] // "unknown"))] | unique | join("; ")')" 207 NODE_DETAIL="system=${SYSTEM_COUNT}, compute=${COMPUTE_COUNT}, gpu=${GPU_COUNT}${GPU_DETAILS:+ (${GPU_DETAILS})}" 208 if [[ "$NOT_READY" -gt 0 ]]; then 209 record nodes FAIL "$NODE_DETAIL; not-ready=${NOT_READY}" 210 elif [[ "$SYSTEM_COUNT" -eq 0 ]]; then 211 record nodes FAIL "$NODE_DETAIL; missing node-type=system" 212 elif [[ "$GPU_COUNT" -eq 0 && "$ALLOW_GPU_ZERO" == "true" ]]; then 213 record nodes WARN "$NODE_DETAIL; no GPU nodes and ALLOW_GPU_ZERO=true" 214 elif [[ "$GPU_COUNT" -eq 0 ]]; then 215 record nodes FAIL "$NODE_DETAIL; missing node-type=gpu" 216 else 217 record nodes PASS "$NODE_DETAIL" 218 fi 219 220 # --- check: ecr-pull --- 221 ECR_REPO_JSON="$(aws ecr describe-repositories --region "$AWS_REGION" \ 222 --query "repositories[?starts_with(repositoryName, '${ECR_REPOSITORY_PREFIX}/')].{name:repositoryName,arn:repositoryArn}" \ 223 --output json 2>/dev/null || echo '[]')" 224 ECR_REPOS="$(echo "$ECR_REPO_JSON" | jq -r '.[].name' | paste -sd ' ' -)" 225 mapfile -t ECR_REPO_ARNS < <(echo "$ECR_REPO_JSON" | jq -r '.[].arn') 226 ECR_REPO_COUNT="${#ECR_REPO_ARNS[@]}" 227 if [[ "${#NODE_ROLE_ARNS[@]}" -eq 0 ]]; then 228 record ecr-pull WARN "no EKS node role ARNs found; cannot prove ECR pull access" 229 else 230 ECR_AUTH_ACTION=(ecr:GetAuthorizationToken) 231 ECR_REPO_ACTIONS=(ecr:BatchCheckLayerAvailability ecr:GetDownloadUrlForLayer ecr:BatchGetImage) 232 all_allowed=true 233 sim_available=true 234 for role_arn in "${NODE_ROLE_ARNS[@]}"; do 235 auth_sim_json="$(aws iam simulate-principal-policy \ 236 --policy-source-arn "$role_arn" \ 237 --action-names "${ECR_AUTH_ACTION[@]}" \ 238 --resource-arns "*" \ 239 --output json 2>/dev/null || true)" 240 repo_sim_json="" 241 if [[ "${#ECR_REPO_ARNS[@]}" -gt 0 ]]; then 242 repo_sim_json="$(aws iam simulate-principal-policy \ 243 --policy-source-arn "$role_arn" \ 244 --action-names "${ECR_REPO_ACTIONS[@]}" \ 245 --resource-arns "${ECR_REPO_ARNS[@]}" \ 246 --output json 2>/dev/null || true)" 247 fi 248 if [[ -z "$auth_sim_json" || ( "${#ECR_REPO_ARNS[@]}" -gt 0 && -z "$repo_sim_json" ) ]]; then 249 sim_available=false 250 role_name="${role_arn##*/}" 251 attached="$(aws iam list-attached-role-policies --role-name "$role_name" --query 'AttachedPolicies[].PolicyArn' --output text 2>/dev/null || true)" 252 if [[ "$attached" != *"AmazonEC2ContainerRegistryReadOnly"* && "$attached" != *"AmazonEC2ContainerRegistryPowerUser"* ]]; then 253 all_allowed=false 254 fi 255 else 256 denied="$(echo "$auth_sim_json" | jq '[.EvaluationResults[] | select(.EvalDecision != "allowed")] | length')" 257 if [[ "${#ECR_REPO_ARNS[@]}" -gt 0 ]]; then 258 repo_denied="$(echo "$repo_sim_json" | jq '[.EvaluationResults[] | select(.EvalDecision != "allowed")] | length')" 259 denied=$((denied + repo_denied)) 260 fi 261 [[ "$denied" -gt 0 ]] && all_allowed=false 262 fi 263 done 264 if [[ "$all_allowed" == "true" ]]; then 265 if [[ "$ECR_REPO_COUNT" -gt 0 ]]; then 266 record ecr-pull PASS "ECR read allowed for node role(s); ${ECR_REPO_COUNT} repo(s) under ${ECR_REPOSITORY_PREFIX}/" 267 else 268 record ecr-pull PASS "ECR read allowed for node role(s); no repos yet under ${ECR_REPOSITORY_PREFIX}/" 269 fi 270 elif [[ "$sim_available" == "false" ]]; then 271 record ecr-pull WARN "could not prove ECR read for every node role; iam:SimulatePrincipalPolicy unavailable and managed policy fallback did not match" 272 else 273 record ecr-pull FAIL "one or more node roles lack required ECR read actions" 274 fi 275 fi 276 277 # --- check: s3-rbac --- 278 if [[ -z "${S3_BUCKET_NAME:-}" ]]; then 279 record s3-rbac WARN "Skipped (no S3_BUCKET_NAME)" 280 elif [[ "${#NODE_ROLE_ARNS[@]}" -eq 0 && -z "${S3_PROBE_ROLE_ARN:-}" ]]; then 281 record s3-rbac WARN "S3_BUCKET_NAME set, but no role ARN is available for IAM simulation" 282 else 283 S3_ROLE_ARN="${S3_PROBE_ROLE_ARN:-${NODE_ROLE_ARNS[0]}}" 284 s3_sim="$(aws iam simulate-principal-policy \ 285 --policy-source-arn "$S3_ROLE_ARN" \ 286 --action-names s3:ListBucket s3:GetObject \ 287 --resource-arns "arn:aws:s3:::${S3_BUCKET_NAME}" "arn:aws:s3:::${S3_BUCKET_NAME}/*" \ 288 --output json 2>/dev/null || true)" 289 if [[ -z "$s3_sim" ]]; then 290 record s3-rbac WARN "Could not simulate S3 access for $S3_ROLE_ARN" 291 else 292 denied="$(echo "$s3_sim" | jq '[.EvaluationResults[] | select(.EvalDecision != "allowed")] | length')" 293 if [[ "$denied" -eq 0 ]]; then 294 record s3-rbac PASS "S3 List/Get allowed on $S3_BUCKET_NAME for ${S3_ROLE_ARN##*/}" 295 else 296 record s3-rbac FAIL "S3 List/Get not fully allowed on $S3_BUCKET_NAME for ${S3_ROLE_ARN##*/}" 297 fi 298 fi 299 fi 300 301 # --- check: gpu-op --- 302 PODS_JSON="$(kubectl --context "$CLUSTER_NAME" get pods -n gpu-operator -o json 2>/dev/null || echo '{"items":[]}')" 303 GPU_OP_TOTAL="$(echo "$PODS_JSON" | jq '.items | length')" 304 GPU_OP_NOT_READY="$(echo "$PODS_JSON" | jq '[.items[] | select(.status.phase != "Succeeded") | select(((.status.conditions // []) | map(select(.type=="Ready" and .status=="True")) | length) == 0)] | length')" 305 GPU_OP_READY=$((GPU_OP_TOTAL - GPU_OP_NOT_READY)) 306 CHART_VER="$(helm --kube-context "$CLUSTER_NAME" list -n gpu-operator -o json 2>/dev/null | jq -r '.[] | select(.name=="gpu-operator") | .chart // ""' | sed 's/^gpu-operator-v\?//' || true)" 307 if [[ "$GPU_OP_TOTAL" -eq 0 ]]; then 308 record gpu-op FAIL "no pods found in gpu-operator namespace" 309 elif [[ "$GPU_OP_NOT_READY" -eq 0 ]]; then 310 if [[ -n "${GPU_OPERATOR_VERSION:-}" && -n "$CHART_VER" && "$CHART_VER" != "${GPU_OPERATOR_VERSION#v}" ]]; then 311 record gpu-op WARN "${GPU_OP_READY}/${GPU_OP_TOTAL} pods Ready; chart v${CHART_VER}, expected v${GPU_OPERATOR_VERSION#v}" 312 else 313 record gpu-op PASS "${GPU_OP_READY}/${GPU_OP_TOTAL} pods Ready${CHART_VER:+; chart v$CHART_VER}" 314 fi 315 else 316 record gpu-op FAIL "${GPU_OP_READY}/${GPU_OP_TOTAL} pods Ready" 317 fi 318 319 # --- check: gpu-resource --- 320 GPU_RESOURCE_NODES="$(echo "$NODES_JSON" | jq '[.items[] | select((.status.allocatable["nvidia.com/gpu"] // "0" | tonumber) >= 1)] | length')" 321 GPU_RESOURCE_TOTAL="$(echo "$NODES_JSON" | jq '[.items[] | (.status.allocatable["nvidia.com/gpu"] // "0" | tonumber)] | add // 0')" 322 if [[ "$GPU_RESOURCE_NODES" -gt 0 ]]; then 323 record gpu-resource PASS "${GPU_RESOURCE_NODES} node(s) advertise nvidia.com/gpu=${GPU_RESOURCE_TOTAL}" 324 elif [[ "$ALLOW_GPU_ZERO" == "true" ]]; then 325 record gpu-resource WARN "no nodes advertise nvidia.com/gpu and ALLOW_GPU_ZERO=true" 326 else 327 record gpu-resource FAIL "no nodes advertise nvidia.com/gpu" 328 fi 329 330 # --- check: cuda-smoke --- 331 SMOKE_POD="aws-validate-cuda-smoke-$$" 332 cleanup_smoke() { 333 kubectl --context "$CLUSTER_NAME" delete pod "$SMOKE_POD" --ignore-not-found >/dev/null 2>&1 || true 334 } 335 if [[ "$SKIP_CUDA" == "true" ]]; then 336 record cuda-smoke WARN "Skipped (SKIP_CUDA=true)" 337 elif [[ "$GPU_RESOURCE_NODES" -eq 0 ]]; then 338 record cuda-smoke WARN "Skipped (no GPU nodes available)" 339 else 340 trap cleanup_smoke EXIT 341 kubectl --context "$CLUSTER_NAME" run "$SMOKE_POD" \ 342 --image="$CUDA_IMAGE" \ 343 --restart=Never \ 344 --overrides='{"spec":{"nodeSelector":{"node-type":"gpu"},"tolerations":[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}],"containers":[{"name":"smoke","image":"'"$CUDA_IMAGE"'","command":["nvidia-smi"],"resources":{"limits":{"nvidia.com/gpu":"1"}}}]}}' \ 345 >/dev/null 2>&1 346 end=$((SECONDS + TIMEOUT_MIN * 60)) 347 phase="" 348 while (( SECONDS < end )); do 349 phase="$(kubectl --context "$CLUSTER_NAME" get pod "$SMOKE_POD" -o jsonpath='{.status.phase}' 2>/dev/null || true)" 350 [[ "$phase" == "Succeeded" || "$phase" == "Failed" ]] && break 351 sleep 5 352 done 353 354 if [[ "$phase" == "Succeeded" ]]; then 355 GPU_MODEL="$(kubectl --context "$CLUSTER_NAME" logs "$SMOKE_POD" 2>/dev/null | grep -Eom1 'A10|L40S|T4|H100|A100|V100|L4' || true)" 356 record cuda-smoke PASS "nvidia-smi pod returned 0${GPU_MODEL:+; $GPU_MODEL detected}" 357 else 358 detail="phase=${phase:-UNKNOWN}" 359 pod_log="$(kubectl --context "$CLUSTER_NAME" logs "$SMOKE_POD" 2>/dev/null | tail -20 | tr '\n' ' ' || true)" 360 [[ -n "$pod_log" ]] && detail="$detail; logs: $pod_log" 361 record cuda-smoke FAIL "nvidia-smi pod did not complete successfully ($detail)" 362 fi 363 cleanup_smoke 364 trap - EXIT 365 fi 366 367 # --- check: s3-probe --- 368 S3_POD="aws-validate-s3-probe-$$" 369 cleanup_s3_probe() { 370 kubectl --context "$CLUSTER_NAME" delete pod "$S3_POD" -n "$S3_PROBE_NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true 371 } 372 if [[ "$SKIP_S3_PROBE" == "true" || -z "${S3_BUCKET_NAME:-}" || -z "${S3_PROBE_SERVICE_ACCOUNT:-}" ]]; then 373 record s3-probe WARN "Skipped (SKIP_S3_PROBE=${SKIP_S3_PROBE} S3_BUCKET_NAME=${S3_BUCKET_NAME:-unset} S3_PROBE_SERVICE_ACCOUNT=${S3_PROBE_SERVICE_ACCOUNT:-unset})" 374 else 375 trap cleanup_s3_probe EXIT 376 kubectl --context "$CLUSTER_NAME" run "$S3_POD" -n "$S3_PROBE_NAMESPACE" \ 377 --image="$S3_PROBE_IMAGE" \ 378 --restart=Never \ 379 --overrides='{"spec":{"serviceAccountName":"'"$S3_PROBE_SERVICE_ACCOUNT"'","containers":[{"name":"aws","image":"'"$S3_PROBE_IMAGE"'","command":["aws","s3api","list-objects-v2","--bucket","'"$S3_BUCKET_NAME"'","--max-items","1"]}]}}' \ 380 >/dev/null 2>&1 381 if kubectl --context "$CLUSTER_NAME" wait pod "$S3_POD" -n "$S3_PROBE_NAMESPACE" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${TIMEOUT_MIN}m" >/dev/null 2>&1; then 382 record s3-probe PASS "serviceAccount=$S3_PROBE_SERVICE_ACCOUNT listed $S3_BUCKET_NAME" 383 else 384 phase="$(kubectl --context "$CLUSTER_NAME" get pod "$S3_POD" -n "$S3_PROBE_NAMESPACE" -o jsonpath='{.status.phase}' 2>/dev/null || echo unknown)" 385 record s3-probe FAIL "S3 probe failed (phase=$phase serviceAccount=$S3_PROBE_SERVICE_ACCOUNT bucket=$S3_BUCKET_NAME)" 386 fi 387 cleanup_s3_probe 388 trap - EXIT 389 fi 390 391 # --- emit report --- 392 printf '{ "cluster": "%s", "timestamp": "%s", "summary": { "pass": %d, "warn": %d, "fail": %d }, "checks": [%s] }\n' \ 393 "$CLUSTER_NAME" "$(date -u +%FT%TZ)" "$PASS" "$WARN" "$FAIL" "$(IFS=,; echo "${CHECKS[*]}")" \ 394 | jq . > "$REPORT_FILE" 395 396 echo "" 397 echo "=== Summary: PASS=$PASS WARN=$WARN FAIL=$FAIL ===" 398 echo "Report: $REPORT_FILE" 399 400 [[ "$FAIL" -gt 0 ]] && exit 1 || exit 0 401 ``` 402 403 ## Validation Checklist 404 405 - [ ] Report uses Azure-compatible `cluster`, `timestamp`, `summary`, `checks[].id`, `checks[].status`, and `checks[].detail` fields. 406 - [ ] Preflight exits with code 2 if a tool is missing or the cluster is unreachable. 407 - [ ] Account check validates `AWS_ACCOUNT_ID` when supplied. 408 - [ ] VPC check reports the EKS VPC CIDR and attached subnet names/CIDRs. 409 - [ ] EKS check requires `ACTIVE` and OIDC issuer. 410 - [ ] Node check verifies `node-type=system` and `node-type=gpu`; `ALLOW_GPU_ZERO=true` downgrades missing GPU nodes to WARN. 411 - [ ] ECR pull check proves required read actions with IAM simulation when possible, with managed-policy fallback. 412 - [ ] Optional S3 RBAC and S3 probe checks WARN when not configured. 413 - [ ] GPU Operator check inspects pod Ready conditions, not just pod phase. 414 - [ ] CUDA smoke pod requests one GPU, waits for `Succeeded`, and is deleted on every exit path. 415 - [ ] No secret values, kubeconfig contents, ECR passwords, or AWS credentials are printed.