NVIDIA GPU Operator on Kubernetes Hardening
Purpose
Review NVIDIA GPU Operator deployments on Kubernetes against NVIDIA documentation and Kubernetes pod-security hardening: device plugin posture, MIG manager configuration, time-sliced GPU configuration, NFD label usage, container toolkit isolation, securityContext posture for GPU workloads, and admission policy enforcement of GPU resource requests.
Lean operating rules
- Prefer live evidence (
kubectl -n gpu-operator get pods,kubectl get clusterpolicy,kubectl get nodes -L nvidia.com/gpu.product, MIG profile annotations) when the active client exposes it; otherwise fall back to NVIDIA GPU Operator documentation and sanitized manifests. - Separate confirmed facts from inference. If MIG strategy, time-slicing config, or admission posture was not directly queried, say so.
- Treat GPU workload pods running with
privileged: trueoutside of the GPU Operator's own DaemonSets as a critical finding - privilege creep across tenant workloads. - Treat MIG-capable nodes running in
singlestrategy on multi-tenant clusters whenmixedis required as a high finding - partition diversity is impossible. - Treat time-sliced GPU configuration shared across tenants without namespace-scoped admission as a high finding - noisy-neighbor and side-channel risk.
- Treat absence of an admission policy (Kyverno / OPA / ValidatingAdmissionPolicy) gating
nvidia.com/gpurequests by namespace as a high finding for multi-tenant clusters. - Treat default GPU Operator deployment with public, unsigned image pulls as a medium finding - add image-verification policy.
- Treat node-feature-discovery labels absent on GPU nodes as a low finding - scheduling reliability is reduced.
Response minimum
Return, at minimum:
- the scoped target (cluster, GPU Operator version, MIG strategy) and evidence level,
- device plugin and toolkit posture,
- MIG / time-slicing posture,
- admission policy posture for GPU resources,
- namespace tenancy posture,
- image verification posture,
- safe next actions and assumptions or blockers.