Eks Gpu Operator

NVIDIA GPU Operator On EKS

What This Skill Produces

Generate generated/<cluster-name>-gpu-operator.sh, a bash script that installs or upgrades NVIDIA GPU Operator on an existing EKS cluster.

Terraform must not install GPU Operator. Keep the Helm release outside Terraform state.

Prerequisites

• kubeconfig context works for CLUSTER_NAME
• GPU node group exists, or the user accepts that validation may warn while it is scaled to zero
• GPU nodes use ami_type = "AL2023_x86_64_NVIDIA" or a reviewed custom GPU AMI with driver and container toolkit already installed

Inputs

EKS Rules

The AWS infra path uses EKS GPU-optimized AMIs. Those AMIs already provide the NVIDIA driver and container toolkit, so install GPU Operator with:

driver.enabled=false
toolkit.enabled=false
operator.defaultRuntime=containerd

GPU Operator still installs Kubernetes GPU operands such as device plugin, DCGM, node feature discovery, and validators.

Bash Script

Generate this script shape:

#!/bin/bash
set -euo pipefail

: "${CLUSTER_NAME:?Set CLUSTER_NAME before running}"
GPU_OPERATOR_VERSION="${GPU_OPERATOR_VERSION:-25.3.1}"

kubectl --context "$CLUSTER_NAME" get nodes >/dev/null

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia --force-update
helm repo update nvidia

tmpdir="$(mktemp -d)"
cleanup() {
  rm -rf "$tmpdir"
}
trap cleanup EXIT

helm pull nvidia/gpu-operator \
  --version "$GPU_OPERATOR_VERSION" \
  --untar \
  --untardir "$tmpdir"

kubectl --context "$CLUSTER_NAME" create namespace gpu-operator \
  --dry-run=client -o yaml | kubectl --context "$CLUSTER_NAME" apply -f -

kubectl --context "$CLUSTER_NAME" apply -f "$tmpdir/gpu-operator/crds"

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --kube-context "$CLUSTER_NAME" \
  --namespace gpu-operator \
  --version "$GPU_OPERATOR_VERSION" \
  --set operator.defaultRuntime=containerd \
  --set driver.enabled=false \
  --set toolkit.enabled=false \
  --wait \
  --timeout 20m

Teardown

For stack destroy, remove GPU Operator before Terraform destroys the cluster:

helm uninstall gpu-operator \
  --kube-context "$CLUSTER_NAME" \
  --namespace gpu-operator \
  --wait --timeout 5m || true
kubectl --context "$CLUSTER_NAME" delete namespace gpu-operator --ignore-not-found

Troubleshooting

Validation Checklist

• Helm release is installed outside Terraform state.
• driver.enabled=false and toolkit.enabled=false are set.
• CRDs are applied before Helm install/upgrade.
• Script uses a unique temporary directory and cleans it up.
• Teardown removes the Helm release before Terraform destroy.