Deploy KAI Scheduler or Run:ai (optional)
Run commands from the DeepOps repository root. This adds GPU-aware scheduling
on top of a working Kubernetes GPU cluster. It is off by default and never changes
the GPU Operator or the NVIDIA driver. Full guide: docs/k8s-cluster/runai.md.
Preconditions
python3 scripts/validation/validate_k8s.py --json --cuda-smokereports"ok": true(the cluster already runs GPU pods). If not, useskills/deploy-k8s-gpu-cluster/first.- Confirm the inventory is the intended cluster and nobody else's workloads run there; the install adds cluster-wide admission webhooks.
- Pick one mode.
kaiandplatformexclude each other (the platform ships its own KAI).platformis experimental and also needs a default StorageClass,kube-prometheus-stack88.6.5+, a DNS name and TLS (see the guide). - Fractioning with hard memory limits (
runai_kai_gpu_fractioning: true) needs GPU Operator v26.7.1+ and an r615+ driver already in place. Do not change the default driver to get it without the cluster owner's agreement.
Procedure
Set
runai_mode(and, for platform, the domain, admin user and a vaulted login value) inconfig/group_vars/k8s_cluster.yml.Preview:
ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml --check --diffApply:
ansible-playbook -l kube_control_plane playbooks/k8s-cluster/runai.yml. Rerun once; the second run must reportchanged=0.Verify with the validator, not with the playbook result:
python3 scripts/validation/validate_runai.py --json # kai python3 scripts/validation/validate_runai.py --json --memory-enforcement # + fractioningRequire exit
0and"ok": truewithgpu_job_okandfraction_job_oktrue. Platform mode: add--mode platform --platform-domain <fqdn>, setRUNAI_USERNAME/RUNAI_PASSWORDin the environment (never as arguments), and use a Run:ai project's namespace and queue.Remove: add
-e runai_state=absent, thenpython3 scripts/validation/validate_runai.py --json --verify-uninstalledmust report"ok": true.
Failure branches
| Signal | Do |
|---|---|
| Role stops with "No node advertises nvidia.com/gpu" | GPU stack not ready; wait, or run the GPU Operator playbook. Do not bypass with runai_require_gpu: false on a real install. |
| Role stops with "ships its own KAI" | Remove the other mode first (runai_state: absent). |
Role stops on GPU Operator version or r615 |
Missing fractioning prerequisites. Set runai_kai_gpu_fractioning: false, or ask the cluster owner to approve the GPU stack change. |
Validator scheduler deployments missing/not available |
KAI operator still starting; wait and rerun. kubectl -n kai-scheduler get pods. |
gpu_job_ok false, pod Pending |
Wrong queue or none: kubectl get queues.scheduling.run.ai; pod must set schedulerName: kai-scheduler and the queue label. |
fraction_job_ok false: pods on different GPUs |
The node has more than one GPU and KAI chose another; with one GPU this means sharing is off (global.gpuSharingMode). |
memory_enforcement_ok false |
Limits are not enforced: check the GPU Operator CDI/NRI settings and kubectl get gpufractioningconfig default -o yaml. Do not claim enforcement. |
| Platform: control plane unreachable | The domain must resolve to the ingress address; pass --ca-file/--resolve-ip. |
| Uninstall check lists leftovers | Remove the named namespaces/releases; add -e runai_uninstall_remove_crds=true only if queues/projects may be deleted. |
Do not
- Install standalone KAI beside the Run:ai cluster chart.
- Put the admin login value in a command line, an inventory file in Git, or a log.
- Treat a finished playbook as success; only the validator's
"ok": truecounts. - Claim a live result you did not run. Offline tests prove the logic, not a GPU job.