Validate a GPU cluster
Which validator
- Slurm cluster →
scripts/validation/validate_slurm.py(run on a cluster node) - Kubernetes cluster →
scripts/validation/validate_k8s.py(run whereverkubectlreaches the cluster) - Not deployed yet / checking the provisioning environment →
scripts/validation/deepops_doctor.py(run from the repository root)
All tools support --json and exit 0 only when every check passes. Full
contract: docs/deepops/validation.md.
Procedure
Run the matching validator with
--json.python3 scripts/validation/validate_slurm.py --json python3 scripts/validation/validate_k8s.py --json --cuda-smokeUse
--skip-gpu-job(Slurm) or omit--cuda-smoke(Kubernetes) when a strictly read-only check is required — for example on a busy production cluster where a test job would queue behind real work.Read the verdict from the JSON, not from ad-hoc commands:
ok: true— report healthy, include the key counts (nodes_total,gpus_configured/gpus_allocatable).ok: false— report each entry infailuresverbatim; they name the failing subsystem and the next diagnostic step.- Use the name-sorted
nodeslist to identify individual failures. Slurm entries reportname, normalizedstate, andgpus_configured; Kubernetes entries reportname,ready, andgpus_allocatable.
When a GPU check fails, do not conclude "driver broken" from a bare
nvidia-smiover SSH — on Slurm nodes GPUs are hidden outside jobs. Followskills/diagnose-driver-install/instead.
Interpreting common results
| Signal | Meaning |
|---|---|
Slurm nodes_unavailable > 0 |
Node down/drained — scontrol show node <name> for the reason. |
Slurm gpus_configured: 0 |
GRES not configured — check config/group_vars/slurm-cluster.yml GPU settings and rerun the Slurm playbook. |
K8s gpus_allocatable: 0 |
GPU Operator stack not ready — kubectl get pods -A | grep -i nvidia; first-deploy driver builds can take 10+ minutes. |
K8s smoke pod Pending |
No schedulable GPU or image pull failure — kubectl -n deepops-validate describe pod. |
Direct nvidia-smi over SSH says "No devices were found" on a Slurm node |
Expected GPU-hide behavior, not a failure. |