Deploy a Kubernetes GPU cluster
Preconditions
- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver installs may reboot them; no active users or workloads).
- SSH access from the provisioning machine to every host as a sudo-capable user.
submodules/kubesprayinitialized — Kubernetes playbooks fail on missingkubespray_defaultsrole imports without it.- Run everything from the repository root.
Procedure
Prepare the environment and verify it:
git submodule update --init --recursive ./scripts/setup.sh cp -r config.example configEdit
config/inventory: control plane nodes under[kube_control_plane]and[etcd], workers under[kube_node](a single machine can hold all three roles).Preflight — must pass before deploying:
python3 scripts/validation/deepops_doctor.py --remote --jsonDeploy:
ansible-playbook -l k8s_cluster playbooks/k8s-cluster.ymlThis runs Kubespray and installs the NVIDIA GPU Operator. Expect roughly 45–90 minutes on a first run.
Validate — the success signal is this, not the play recap:
python3 scripts/validation/validate_k8s.py --json --cuda-smokeRequire
"ok": truewithnodes_ready == nodes_total,gpus_allocatable > 0, andcuda_smoke_ok: true.
Failure branches
- Playbook fails on a transient error: rerun the same playbook;
Kubespray is rerun-safe. A converged rerun reports
changed=0. - Syntax/import error mentioning
kubespray_defaults: the submodule is not initialized; rungit submodule update --init --recursive. gpus_allocatable: 0: the GPU Operator stack is not ready. Checkkubectl get pods -A | grep -i nvidia— the driver DaemonSet can take 10+ minutes on first deploy; if pods are crash-looping, followskills/diagnose-driver-install/.- CUDA smoke pod stuck
Pending:kubectl -n deepops-validate describe pod deepops-validate-cuda— usually no allocatable GPU (see above) or an image pull problem on airgapped networks (use--cuda-imageto point at a mirrored image). - Single-node clusters: control plane taints are handled by the
playbook for the single-node case; if pods stay Pending on a multi-role
node, check taints with
kubectl describe node <name> | grep -i taint.