Aks Nodes

AKS additional node pools (GPU + compute)

What this skill produces

Adds 0–2 additional node pools to an existing AKS cluster:

GPU pool MUST be single-zone if LLS will be deployed — Azure Standard LB zonality must match the GPU pool's zone (see nvcf-azure-deploy architecture rule #9 — or locally tf-conventions).

GPU SKU recommendations (validate quota per region; quota requests take 24–72h):

Inputs

Source from streaming-env.sh + generated/.env.iam-vpc + generated/.env.aks-cluster.

Required:

Optional (firm defaults):

Outputs

• Bash: generated/stage03-<cluster-name>-aks-nodes.sh — idempotent (az aks nodepool add errors on exists; the script checks first).
• Terraform: generated/<cluster-name>/aks-nodes/main.tf containing azurerm_kubernetes_cluster_node_pool resources.

GPU driver source — pick one

AKS NV/NC SKUs pre-install the NVIDIA driver. The GPU Operator (from the gpu-operator skill) also wants to install one. Pick exactly one source:

For default validation, leave at enabled.

Bash snippet (reference)

#!/bin/bash
set -euo pipefail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
for envfile in .env.iam-vpc .env.aks-cluster; do
  if [[ ! -f "$REPO_ROOT/generated/$envfile" ]]; then
    echo "ERROR: $REPO_ROOT/generated/$envfile not found. Run earlier stages first." >&2
    exit 1
  fi
  source "$REPO_ROOT/generated/$envfile"
done

: "${CLUSTER_NAME:?}" "${RESOURCE_GROUP:?}" "${NODE_SUBNET_ID:?}"
GPU_POOL_NAME="${GPU_POOL_NAME:-gpu}"
GPU_VM_SIZE="${GPU_VM_SIZE:-Standard_NV36ads_A10_v5}"
GPU_NODE_COUNT="${GPU_NODE_COUNT:-0}"
GPU_MIN_COUNT="${GPU_MIN_COUNT:-0}"
GPU_MAX_COUNT="${GPU_MAX_COUNT:-2}"
GPU_ZONES="${GPU_ZONES:-1}"
GPU_NODE_TAINTS="${GPU_NODE_TAINTS:-}"   # empty = no taint (AWS-parity); label-only GPU-pool reservation
GPU_DRIVER_INSTALL="${GPU_DRIVER_INSTALL:-enabled}"
ENABLE_COMPUTE_POOL="${ENABLE_COMPUTE_POOL:-false}"

az account set --subscription "$AZURE_SUBSCRIPTION_ID"

# Quota preflight — surface failures BEFORE the slow nodepool add.
# Derive a 2-char UPPERCASE prefix from the SKU family (e.g. Standard_NV36ads_A10_v5 → "NV")
# because `az vm list-usage` returns uppercase quota names.
echo "--- Quota check for $GPU_VM_SIZE in $LOCATION ---"
SKU_PREFIX=$(echo "$GPU_VM_SIZE" | awk -F_ '{print toupper(substr($2,1,2))}')
az vm list-usage --location "$LOCATION" \
  --query "[?contains(name.value, '$SKU_PREFIX')].{name:name.localizedValue, limit:limit, current:currentValue}" \
  -o table || echo "WARN: quota lookup failed; proceeding anyway"

# GPU pool — idempotent add
if az aks nodepool show --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$GPU_POOL_NAME" &>/dev/null; then
  echo "GPU pool $GPU_POOL_NAME already exists; skipping add"
else
  DRIVER_FLAG=""
  if [[ "$GPU_DRIVER_INSTALL" == "disabled" ]]; then
    DRIVER_FLAG="--skip-gpu-driver-install"
  fi
  # Taint is opt-in and empty by default (AWS-parity). Only pass --node-taints when set,
  # otherwise the pool is reserved by the nvidia.com/gpu=true label alone.
  TAINT_FLAG=""
  if [[ -n "$GPU_NODE_TAINTS" ]]; then
    TAINT_FLAG="--node-taints $GPU_NODE_TAINTS"
  fi
  echo "--- Adding GPU node pool $GPU_POOL_NAME ($GPU_VM_SIZE, zone $GPU_ZONES) ---"
  az aks nodepool add \
    --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" \
    --name "$GPU_POOL_NAME" --node-vm-size "$GPU_VM_SIZE" \
    --node-count "$GPU_NODE_COUNT" \
    --enable-cluster-autoscaler --min-count "$GPU_MIN_COUNT" --max-count "$GPU_MAX_COUNT" \
    --zones $GPU_ZONES \
    $TAINT_FLAG \
    --labels nvidia.com/gpu=true \
    --vnet-subnet-id "$NODE_SUBNET_ID" \
    $DRIVER_FLAG \
    --only-show-errors
fi

# Optional compute pool
if [[ "$ENABLE_COMPUTE_POOL" == "true" ]]; then
  COMPUTE_POOL_NAME="${COMPUTE_POOL_NAME:-compute}"
  if az aks nodepool show --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$COMPUTE_POOL_NAME" &>/dev/null; then
    echo "Compute pool $COMPUTE_POOL_NAME already exists; skipping add"
  else
    echo "--- Adding compute node pool $COMPUTE_POOL_NAME ---"
    az aks nodepool add \
      --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" \
      --name "$COMPUTE_POOL_NAME" \
      --node-vm-size "${COMPUTE_VM_SIZE:-Standard_D8ds_v5}" \
      --node-count "${COMPUTE_MIN_COUNT:-1}" \
      --enable-cluster-autoscaler --min-count "${COMPUTE_MIN_COUNT:-1}" --max-count "${COMPUTE_MAX_COUNT:-1}" \
      --zones "${COMPUTE_ZONES:-1}" \
      --labels "${COMPUTE_NODE_LABELS:-node-type=compute}" \
      --vnet-subnet-id "$NODE_SUBNET_ID" \
      --only-show-errors
  fi
fi

kubectl get nodes -o wide

Terraform snippet (reference)

resource "azurerm_kubernetes_cluster_node_pool" "gpu" {
  name                  = var.gpu_pool_name
  kubernetes_cluster_id = var.cluster_id
  vm_size               = var.gpu_vm_size
  node_count            = var.gpu_node_count
  auto_scaling_enabled  = true
  min_count             = var.gpu_min_count
  max_count             = var.gpu_max_count
  zones                 = var.gpu_zones
  vnet_subnet_id        = var.node_subnet_id
  node_labels           = { "nvidia.com/gpu" = "true" }
  # No taint by default — label-only reservation, matching the AWS foundation.
  # A sku=gpu:NoSchedule taint blocks NVCF function pods (no matching toleration) → capacity:0.
  # Set node_taints only if you also wire matching tolerations onto every GPU workload.
  node_taints           = var.gpu_node_taints   # declare with default [] in your variables.tf

  # GPU_DRIVER_INSTALL = "disabled" needs an azapi resource (not exposed by azurerm today):
  #   resource "azapi_update_resource" "skip_driver" {
  #     type        = "Microsoft.ContainerService/managedClusters/agentPools@2024-02-01"
  #     resource_id = azurerm_kubernetes_cluster_node_pool.gpu.id
  #     body = jsonencode({ properties = { gpuProfile = { installGPUDriver = false } } })
  #   }
}

resource "azurerm_kubernetes_cluster_node_pool" "compute" {
  count = var.enable_compute_pool ? 1 : 0

  name                  = var.compute_pool_name
  kubernetes_cluster_id = var.cluster_id
  vm_size               = var.compute_vm_size
  auto_scaling_enabled  = true
  min_count             = var.compute_min_count
  max_count             = var.compute_max_count
  zones                 = var.compute_zones
  vnet_subnet_id        = var.node_subnet_id
  node_labels           = { "node-type" = "compute" }
}

Teardown

az aks nodepool delete --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$GPU_POOL_NAME" --no-wait
az aks nodepool delete --cluster-name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --name "$COMPUTE_POOL_NAME" --no-wait

Validation checklist

• GPU pool name is 1–12 chars, lowercase alphanumeric (no hyphens) — AKS hard constraint
• Quota preflight runs before nodepool add (skippable but should print a warning, not silently bypass)
• Quota query uses an uppercase SKU prefix (e.g. NV, NC) — az vm list-usage returns uppercase quota names; lowercase prefix never matches
• GPU pool is single-zone if LLS will be deployed
• nvidia.com/gpu=true label is set on the GPU pool
• GPU pool has no taint by default (label-only reservation, matching the AWS foundation). A taint is opt-in via GPU_NODE_TAINTS and must be paired with matching workload tolerations — otherwise NVCF function pods can't schedule (NVCA capacity: 0)
• GPU_DRIVER_INSTALL=disabled path uses --skip-gpu-driver-install (and a TF azapi snippet is provided as a comment)
• Compute pool is gated by ENABLE_COMPUTE_POOL=true
• Both pools attach to NODE_SUBNET_ID from iam-vpc
• Idempotent (check before add)
• --teardown removes both pools

References