Deploy a Slurm GPU cluster
Preconditions
- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver installs may reboot them; no active users or workloads).
- SSH access from the provisioning machine to every host as a sudo-capable user.
- Run everything from the repository root.
Procedure
Prepare the environment and verify it:
git submodule update --init --recursive ./scripts/setup.sh cp -r config.example configEdit
config/inventory: put the controller under[slurm-master]and compute nodes under[slurm-node](a single machine can be both). Set the connection user in[all:vars]if not root.Preflight — must pass before deploying:
python3 scripts/validation/deepops_doctor.py --remote --jsonFix anything in
failures(each check'sdetailsays how) and rerun.Deploy:
ansible-playbook -l slurm-cluster playbooks/slurm-cluster.ymlThis installs NVIDIA drivers, builds and configures Slurm, and sets up munge, NFS, and node health checks. Expect roughly 30–60 minutes on a first run.
Validate on a cluster node — the success signal is this, not the play recap:
python3 scripts/validation/validate_slurm.py --jsonRequire
"ok": truewithgpu_job_ok: trueandnodes_unavailable: 0.
Failure branches
- Playbook fails on a transient error (mirror timeout, apt lock,
network blip): rerun the same playbook; it is idempotent. A converged
rerun ends with
changed=0. nvidia-smiworks in the validator's srun job but "fails" over SSH: that is the login GPU-hide behavior, not an error (see AGENTS.md gotchas).gpu_job_ok: falsewith driver errors: followskills/diagnose-driver-install/.- Node shows
downordrainedinnode_states: checkscontrol show node <name>for the reason; after fixing, resume withscontrol update nodename=<name> state=resume. - Wrong or stale host facts after reprovisioning a node: rerun with
--flush-cache.