# Osdc Runners Nodepools

> OSDC runners, NodePools, BuildKit, GitHub Actions constraints, EKS node taints, image mirroring, and the runner/nodepool change checklist. Applies to ~/meta/ci-infra/osdc. Load when modifying runners, nodepools, BuildKit, or node configurations.

- Skill: `pytorch/osdc-runners-nodepools` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pytorch/osdc-runners-nodepools`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pytorch/osdc-runners-nodepools/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: pytorch (https://skillmd.com/u/pytorch)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pytorch/osdc-runners-nodepools

---


# OSDC Runners, NodePools & BuildKit

## Quality Gates

Before declaring runner/nodepool work complete: run `just lint` and `just test` from `osdc/`. All 13 linters and unit tests must pass — these gates are mandatory per the project CLAUDE.md.

## Runner & NodePool Change Checklist (MANDATORY)

When changing runner definitions (`modules/arc-runners/defs/`, `modules/arc-runners-h100/defs/`, `modules/arc-runners-b200/defs/`) or NodePool definitions (`modules/nodepools/defs/`, `modules/nodepools-h100/defs/`, `modules/nodepools-b200/defs/`), you MUST update the following `scripts/python/` files to stay in sync:

| File | What to update |
|------|---------------|
| `scripts/python/instance_specs.py` | `INSTANCE_SPECS` (vcpu, memory_gib, memory_mi, gpu, arch — used by `analyze_node_utilization.py`, `generate_buildkit.py`, `collect_instance_memory.py`, `simulate_cluster.py`), `ENI_MAX_PODS` (AWS-stock max-pods used by simulators and BuildKit sizing) |
| `scripts/python/pytorch_workload_data.py` | `OLD_TO_NEW_LABEL` (update old->new runner name mappings when names change) |
| `scripts/python/simulate_cluster.py` | Uses `analyze_node_utilization` functions — verify simulation still works |
| `scripts/python/simulate_cluster_cli.py` | CLI entry point for simulation — re-run to validate packing |
| `integration-tests/workflows/integration-test.yaml.tpl` | Update `runs-on` labels if runner names changed. Jobs are wrapped in `# BEGIN_<TAG>` / `# END_<TAG>` markers; if you add a new gate, register it in `TAG_REQUIREMENTS` in `integration-tests/scripts/python/phases.py` so the orchestrator strips jobs on clusters missing the required modules. |
| `docs/runner_naming_convention.md` | Update runner name examples and mapping tables |

**Verification**: After any runner/nodepool change, run `just analyze-utilization` to confirm packing efficiency and `just test` to verify all scripts agree on the new values.

**DaemonSet impact**: When changing runner/job pod resources or adding new instance types, account for per-node DaemonSet overhead. The BuildKit pod-sizing algorithm subtracts a fixed 300m CPU / 440Mi memory budget (`DAEMONSET_OVERHEAD_CPU_M` / `DAEMONSET_OVERHEAD_MEM_MI` in `modules/buildkit/scripts/python/generate_buildkit.py`) before dividing per-node capacity. Cluster-wide base DaemonSets that fit inside this budget (manifests under `base/kubernetes/`): `registry-mirror-config`, `node-performance-tuning`, `algif-mitigation`, `dirtyfrag-mitigation`, `image-cache-janitor`, `nvidia-device-plugin`, plus DCGM on GPU nodes. `nodelocaldns` (25m CPU / 100Mi memory) is under `base/kubernetes/nodelocaldns/` and deployed via its own deploy.sh. Per-module DaemonSets include `cache-enforcer` (module) and `runner-hooks-warmer` (declared in `modules/arc/kubernetes/hooks-warmer.yaml`, runs in the `arc-runners` namespace, but pinned to `node-fleet=c7i-runner` via nodeSelector). Helm/EKS-addon overhead (kube-prometheus-stack node-exporter, Alloy logging, kube-proxy, vpc-cni, ebs-csi-node) is enumerated in `scripts/python/daemonset_overhead.py` (`HELM_DAEMONSETS` + `EKS_ADDON_DAEMONSETS`). The live computed total is available via `uv run scripts/python/daemonset_overhead.py`. New cluster-wide DaemonSets MUST tolerate the standard taints listed in the EKS Node Taints section below.

## GitHub Actions Workflow Constraints

All self-hosted runner pods run in `containerMode: kubernetes-novolume`. Workflow jobs typically need a `container:` image — containerless jobs may be rejected by the runner-container-hooks. Runner image is digest-pinned per OSDC commit by `modules/arc-runners/scripts/python/resolve_runner_version.py` — never use `:latest` and there is no Renovate bot. Runner-container-hooks fork: the `runner-hooks-warmer` DaemonSet (`modules/arc/kubernetes/hooks-warmer.yaml`, deployed by the `arc` module into the `arc-runners` namespace) pins **`v0.8.15`** (https://github.com/jeanschmidt/runner-container-hooks/releases/tag/v0.8.15) — this is the version actually placed on each node. A stale comment in `runner.yaml.tpl` still references `v0.8.13`; the warmer DaemonSet wins. The `wait-for-hooks` init container (Alpine, polls `/mnt/host-hooks/dist/index.js` for up to 300s) snapshots the patched hooks into an emptyDir consumed by every runner pod.

### Runner image resolution (per-commit digest pinning)

`modules/arc-runners/deploy.sh` resolves the runner image in this order:

1. **Operator override**: if `arc.runner_image_tag` is set in `clusters.yaml` (under the cluster's `arc:` block), the deploy uses `ghcr.io/actions/actions-runner:<tag>` (tag-only, no digest). The resolver is bypassed entirely — no GitHub API call, no `crane` call, ConfigMap untouched. Use this as the rollback escape hatch.
2. **Auto-resolve**: otherwise `resolve_runner_version.py` is invoked. It uses the SHA of the most recent commit touching anything under `osdc/` as a lookup key into the `arc-runner-version-lock` ConfigMap (namespace `osdc-system`, key `history.json`, capped at 20 entries newest-first). Cache hit returns the locked `tag@digest`. Cache miss calls GitHub `/releases/latest`, resolves the digest via `crane digest`, prepends the entry, and writes back with optimistic-concurrency (resourceVersion); 409 retries up to 5 times.
3. **Final fallback**: `generate_runners.py` line ~275 contains `cluster_config.get("runner_image", "ghcr.io/actions/actions-runner:2.333.1")` — but in practice `deploy.sh` always exports `RUNNER_IMAGE`, so the literal `2.333.1` only ever appears via unit tests, never via deploy.

The three `arc-runners*` modules (`arc-runners`, `arc-runners-h100`, `arc-runners-b200`) deploy concurrently and share the same lock ConfigMap. The H100 and B200 modules `exec` into the base `arc-runners/deploy.sh` with `ARC_RUNNERS_DEFS_DIR` / `ARC_RUNNERS_OUTPUT_DIR` / `ARC_RUNNERS_MODULE_NAME` overrides, so they inherit the resolver result. See `docs/runner-image-autoresolve.md` for the full ConfigMap shape, failure modes, and rollback paths.

There is no longer a `.github/workflows/osdc-auto-update-deploy-prod.yml` — deploy-time resolution replaced the Renovate flow. The smoke test `modules/arc-runners/tests/smoke/test_runner_version_lock.py` validates the ConfigMap shape and cross-checks the locked entry against the deployed `AutoscalingRunnerSet` pod templates.

## EKS Node Taints

**Base nodes**: `CriticalAddonsOnly=true:NoSchedule`. All base workloads (Harbor, DaemonSets, Karpenter, control plane) must tolerate this.

**Workload (Karpenter) nodes** — runner and GPU NodePools emit these taints:
- `node-fleet=<fleet-name>:NoSchedule` — fleet isolation (e.g. `c7i-runner`, `g5`, `m8g`)
- `instance-type=<instance>:NoSchedule` — per-instance-type taint
- `nvidia.com/gpu=true:NoSchedule` — GPU pools only

BuildKit NodePools are the exception: they emit only `workload/buildkit-<arch>`, no `instance-type` taint (one pool per size, but one shared pod spec).

**IPv6-only EKS**: the cluster runs on IPv6-only pod networking (`ip_family = "ipv6"`, commit `a6b4c8c` / PR #576). Pod IPs are allocated from a /80 IPv6 prefix per node via VPC CNI prefix delegation; the service CIDR is auto-assigned by EKS in `fd00:ec2::/108` (ULA). There is no per-AZ NodePool fan-out, no `bucket` label, no ENIConfig CR, and no `pod_cidr_buckets` map in `clusters.yaml` — the IPv4 + Custom Networking bucket scheme described in the abandoned `INCREASE_IPV4.md` plan was superseded by the IPv6 migration and is NOT in the current generator.

**Startup taints** (cleared by per-DaemonSet init containers via the shared `taint_remover.py` at end-of-init; registry in `modules/nodepools/scripts/python/generate_nodepools.py:STARTUP_TAINTS`):
- `node-init.osdc.io/registry-mirror=true:NoSchedule` — emitted on every cluster; cleared by `registry-mirror-config`
- `node-init.osdc.io/perf-tuning=true:NoSchedule` — emitted on every cluster; cleared by `node-performance-tuning`
- `node-init.osdc.io/algif-mitigation=true:NoSchedule` — emitted on every cluster (TEMPORARY, removed in lockstep with the DaemonSet once AL2023 kernel 6.12.85+ is rolled out); cleared by `algif-mitigation`
- `node-init.osdc.io/dirtyfrag-mitigation=true:NoSchedule` — emitted on every cluster (TEMPORARY, removed in lockstep with the DaemonSet once AL2023 kernel 6.1.170+ or 6.12.83+ is rolled out); cleared by `dirtyfrag-mitigation`
- `node-init.osdc.io/cache-enforcer=true:NoSchedule` — emitted only on clusters with `cache-enforcer` in `NODEPOOLS_ENABLED_MODULES` AND on nodepool defs that pass the per-def `applies_when` predicate. The predicate (`generate_nodepools.py:62-65`) skips release-runner pools (`extra_labels.osdc.io/runner-class == "release"`) because the cache-enforcer DaemonSet has matching `DoesNotExist` nodeAffinity and never schedules there — emitting the taint would deadlock those nodes. Cleared by `cache-enforcer`.

`STARTUP_TAINTS` registry is validated at generator startup (`generate_nodepools.py:_validate_startup_taints_registry`) — each entry's `module` field must match a sibling directory under `modules/`, catching typos before they ship.

Workload pods do NOT tolerate `node-init.osdc.io/*` — they wait for every applicable taint to be removed. Runner pod tolerations are `node-fleet`, `instance-type`, plus `nvidia.com/gpu` on GPU job pods.

## Image Mirroring

For image mirroring details (bootstrap images, ECR, Harbor proxy cache), see the `osdc-harbor` skill. Cluster-wide DaemonSets that pull from already-mirrored upstreams (e.g. `nodelocaldns` from `registry.k8s.io`) do NOT need a per-image pre-mirror entry — the existing `registry-mirror-config` proxy cache handles them lazily on first pull (~30-60s ImagePullBackOff possible on a fresh Karpenter node before containerd `hosts.toml` is written).

## BuildKit Build Service

BuildKit (`moby/buildkit:v0.29.0`) runs as two Deployments in the `buildkit` namespace — one per architecture. Runner job pods invoke `buildctl` — no Kubernetes API access required.

- **Architecture**: Dual-arch fleet with per-arch Deployments and Services
  - `buildkitd-arm64` — Graviton (`m7gd.16xlarge:4` from `defaults.buildkit`), Service: `buildkitd-arm64.buildkit:1234`
  - `buildkitd-amd64` — Intel (`m6id.24xlarge:2` preferred, `m6id.12xlarge:1` fallback, from `defaults.buildkit`), Service: `buildkitd-amd64.buildkit:1234`
  - `buildkitd` — combined Service (round-robin across both arches, for arch-agnostic builds)
- **Sizing**: Dynamically computed by `modules/buildkit/scripts/python/generate_buildkit.py` from instance specs. Guaranteed QoS (requests == limits), static CPU pinning, `max-parallelism=1` (one build at a time per pod), pods per node declared per instance type in `buildkit.{amd64,arm64}_instance_types`
- **NUMA**: BuildKit EC2NodeClasses use `topologyManagerPolicy: restricted` + `topologyManagerScope: container` + `prefer-closest-numa-nodes: true` — stricter than the nodepool generator's `best-effort` default. Relevant when debugging BuildKit scheduling failures on multi-NUMA hosts.
- **Startup taint**: BuildKit nodepools also carry their own `git-cache-not-ready=true:NoSchedule` startup taint (in addition to the cluster-wide `node-init.osdc.io/*` ones). The git-cache rsync init clears it before pods can schedule.
- **Instance types**: Configurable via `clusters.yaml` (`buildkit.arm64_instance_types`, `buildkit.amd64_instance_types`) — a map of instance type to pods per node, in preference order. Each entry generates its own weighted NodePool sharing one EC2NodeClass (`buildkit-<arch>` at weight 100, then `buildkit-<arch>-<type>` descending by 10), so Karpenter falls through to the next size only when the one above cannot provision — e.g. amd64 prefers m6id.24xlarge and spills to m6id.12xlarge on InsufficientInstanceCapacity. Each pool's limits are sized to absorb the whole replica count alone. Both sizes are the same family, so a family-wide regional ICE still stalls; a genuine fallback needs a different NVMe family (`m7id.*`), which the git cache and `instanceStorePolicy: RAID0` both require.
- **Scaling**: Configurable via `clusters.yaml` (`buildkit.replicas_per_arch`, default 12 in `defaults:`; the bash fallback in `deploy.sh` is 4, used only if neither `defaults:` nor the cluster sets a value)
- **Autoscaling (KEDA)**: Enabled via `buildkit.autoscaling.enabled: true`. BuildKit deploys on exactly five clusters — `meta-staging-aws-{uw1,ue1,ue2}` and `meta-prod-aws-{ue1,ue2}` — and all five have it enabled; `defaults.buildkit.autoscaling.enabled` is `false`. Requires the `keda` module to be in the cluster's module list. Per-arch `_min`/`_max`/`_fallback` knobs live under `buildkit.autoscaling.*` in `clusters.yaml`. ScaledObjects are generated by `generate_buildkit.generate_autoscaling_yaml` and applied from `generated/autoscaling.yaml`. With autoscaling on, Deployments omit the static `replicas:` line and add `terminationGracePeriodSeconds: 8100` for in-flight builds.
- **Storage**: NVMe instance storage (RAID0) for build cache + git object cache
- **Registry mirrors**: `buildkitd.toml` routes `FROM` image pulls through Harbor for docker.io, ghcr.io, nvcr.io, registry.k8s.io, quay.io (public.ecr.aws not mirrored — no rate limits)
- **Network access**: NetworkPolicy restricts ingress to pods from `arc-runners` namespace only
- **Load balancing**: HAProxy `buildkitd-lb` (least-connections) distributes `buildctl` connections across buildkitd pods per architecture. Backends are discovered via headless Service DNS (`buildkitd-arm64-pods`, `buildkitd-amd64-pods`, `buildkitd-pods` — `clusterIP: None`) with `resolve-prefer ipv6` on each `server-template` line because under IPv6-only EKS the pod IPs only emit AAAA records.

### Targeting an architecture

```bash
# Build an ARM64 image
buildctl --addr tcp://buildkitd-arm64.buildkit:1234 build --output type=image,name=$IMAGE,push=true ...

# Build an x86_64 image
buildctl --addr tcp://buildkitd-amd64.buildkit:1234 build --output type=image,name=$IMAGE,push=true ...

# Multi-arch: build both, then combine with crane
crane index append -t $IMAGE -m $IMAGE-arm64 -m $IMAGE-amd64
```

### Using the git cache in Dockerfiles

The git-cache rsync runs on BuildKit nodes (same as runner nodes). The buildkitd pod mounts the cache at `/opt/git-cache`. Pass it as a named build context, then bind-mount in the Dockerfile and set `GIT_ALTERNATE_OBJECT_DIRECTORIES`:

```bash
buildctl ... --opt context:gitcache=local:gitcache --local gitcache=/opt/git-cache ...
```

```dockerfile
RUN --mount=type=bind,from=gitcache,source=pytorch/pytorch.git/objects,target=/tmp/git-objects \
    GIT_ALTERNATE_OBJECT_DIRECTORIES=/tmp/git-objects \
    git clone https://github.com/pytorch/pytorch /workspace
```

## Module-Specific Operational Knowledge

### BuildKit Pod Sizing Algorithm

Pod resource requests are computed by `modules/buildkit/scripts/python/generate_buildkit.py` from a static instance spec table. The **first** entry in `{arch}_instance_types` is the primary and alone sizes the pod:
1. Look up total vCPU + memory for the primary instance type
2. Subtract kubelet reserved resources
3. Subtract DaemonSet overhead (300m CPU, 440Mi memory)
4. Apply 10% margin
5. Divide by the primary's `pods_per_node`

Every later entry is then checked with `pods_that_fit`: it must hold at least the count it declared, or generation fails. That is what stops a denser `pods_per_node` on a fallback size from silently shrinking every build pod on the arch — to size the pod off a different entry, make it first.

This ensures exactly N pods fit per node with Guaranteed QoS (requests == limits -> static CPU pinning). NVMe instance storage uses `instanceStorePolicy: RAID0` on EC2NodeClass (nodeadm handles formatting/mounting) — instance types must have `d` suffix.

When adding a new instance type: update BOTH `INSTANCE_SPECS` (vcpu/memory/gpu/arch) and `ENI_MAX_PODS` (AWS-stock max-pods) in `scripts/python/instance_specs.py`.

### EC2NodeClass userData

Generated EC2NodeClass `userData` is multipart MIME. The `application/node.eks.aws` part always carries kubelet config (cpuManagerPolicy, topologyManagerPolicy/Scope, log size/files). Defaults: `cpuManagerPolicy: static`, `topologyManagerPolicy: best-effort`, `topologyManagerScope: container`. BuildKit overrides this to `topologyManagerPolicy: restricted` + `prefer-closest-numa-nodes: true` for tight NUMA placement — it is the only generator that doesn't use `best-effort`. Most pools stop at the kubelet block — registry mirrors, CPU governor, GPU persistence, and the algif_aead / DirtyFrag CVE blacklists are handled by base DaemonSets (`registry-mirror-config`, `node-performance-tuning`, `algif-mitigation`, `dirtyfrag-mitigation`).

### EC2NodeClass metadataOptions — IPv6 IMDS

Generated EC2NodeClass templates set `spec.metadataOptions.httpProtocolIPv6: enabled`. Under IPv6-only EKS the instance metadata service (IMDS) is reachable at `[fd00:ec2::254]` instead of the IPv4 link-local `169.254.169.254`. Workloads, kubelet, and AWS SDKs that talk to IMDS must do so over IPv6; enabling `httpProtocolIPv6` is what makes the IMDS endpoint listen on the IPv6 link-local address. Applied to: every nodepool generator output (`modules/nodepools/scripts/python/generate_nodepools.py`) and the pypi-cache nodepool template (`modules/pypi-cache/kubernetes/ec2nodeclass.yaml.tpl`).

GPU pools (H100, B200) optionally append a `text/x-shellscript` MIME part via the per-def `user_data_script` field — used today for one-shot containerd registry mirror config that must be in place before any image pull. Prefer DaemonSets where possible; reserve `user_data_script` for boot-critical setup only.

**TEMPORARY mitigations** — two kernel-mod blacklist DaemonSets with identical shape and the same lifecycle:
- `base/kubernetes/algif-mitigation.yaml` — modprobe blacklist for CVE-2026-31431 ("Copy Fail" algif_aead LPE). Remove once nodes are on a kernel 6.12.85+ AL2023 AMI. AMI pinnings tagged with `TODO(CVE-2026-31431)` markers are the trigger points: `clusters.yaml`, `modules/nodepools/scripts/python/generate_nodepools.py`, `modules/buildkit/scripts/python/generate_buildkit.py`, `modules/pypi-cache/kubernetes/ec2nodeclass.yaml.tpl`. Watch https://explore.alas.aws.amazon.com/CVE-2026-31431.html.
- `base/kubernetes/dirtyfrag-mitigation.yaml` — modprobe blacklist for CVE-2026-43284 + CVE-2026-43500 ("DirtyFrag" page-cache write LPEs in xfrm-ESP / RxRPC). Also drops the page cache to evict any DirtyFrag-poisoned pages. Same TEMPORARY lifecycle as algif-mitigation.

### NodePool Def File Schema

`modules/nodepools/defs/*.yaml` supports three top-level forms (detected by the generator):

**`nodepool:`** — single instance type (one NodePool from one file). Currently unused; most "single-instance" pools are written as a `fleet:` with one entry instead.
```yaml
nodepool:
  name: example
  instance_type: c7i.16xlarge
  arch: amd64                  # auto-detected if omitted
  node_disk_size: 100
  gpu: true
  has_nvme: true
  topology_manager_policy: single-numa-node   # default best-effort
  topology_manager_scope: pod                 # default container
  baremetal: true                             # uses NODEPOOLS_BAREMETAL_CONSOLIDATE_AFTER (1h)
  exclude_regions: [us-west-1]
  capacity_type: on-demand                    # or "reserved" for Capacity Blocks
  capacity_reservation_ids: [cr-xxxx]         # only with capacity_type: reserved
  user_data_script: scripts/h100-node-setup.sh # optional shell MIME part
  node_compactor: true                        # per-def override of cluster default
```

**`fleet:`** — multi-instance fleet (one NodePool per instance, all sharing a `node-fleet` label/taint). Used by every current def — including single-instance GPU pools (`p4d.yaml`, `nodepools-h100/defs/p5.yaml`, `nodepools-b200/defs/p6.yaml`), which are fleets with one instance entry.
```yaml
fleet:
  name: g5
  arch: amd64
  gpu: true
  exclude_regions: [us-west-1]
  instances:
    - type: g5.48xlarge
      weight: 100              # Karpenter prefers higher weights
      node_disk_size: 600
      has_nvme: true
      baremetal: true          # per-instance optional
```

**`fleets:`** — multi-fleet file (several fleets in one YAML). Detected by the generator but unused in the current defs.

**`-large` companion fleets** (dual-fleet pattern): full-node 8-GPU runners (`l-bx86iavx512-88-1000-a100-8`, `l-bx86iamx-176-1800-h100-8`, `l-bx86iamx-176-1800-b200-8`) override `node_fleet` to point at dedicated `-large` companion pools: `p4d-large`, `p5-large`, `p6-b200-large` (plus CPU equivalents `c7i-large`, `c7a-large`, `r7a-large`, `m7g-metal`, `m8g-large`, `g4dn-metal`, `g5-large`). The companion fleets deliberately do **NOT** set `single-numa-node` because an 8-GPU pod spans both NUMA nodes and would hit `TopologyAffinityError` under `single-numa-node`. The packed multi-tenant pool keeps the strict NUMA policy; the `-large` companion stays at `best-effort`.

NUMA defaults: `topologyManagerPolicy: best-effort` for everything by default. Only the packed GPU pools `p4d`, `p5`, `p6` pin `single-numa-node` with `pod` scope. Their `-large` companions stay at `best-effort` so the whole-node 8-GPU runner can span both NUMA nodes.

**Not in the schema**: there is no `bucket`, `max_pods`, `pod_cidr_buckets`, or per-AZ fan-out. EC2NodeClasses are emitted without `spec.kubelet.maxPods` (kubelet uses the AWS-stock default per `ENI_MAX_PODS`). An IPv4 + Custom Networking bucket scheme was considered (`INCREASE_IPV4.md`) but the codebase moved to IPv6-only EKS in PR #576; do not add these fields — they will be silently ignored by the generator.

Generated YAMLs contain `CLUSTER_NAME_PLACEHOLDER` — `deploy.sh` does `sed` replacement at apply time with the actual cluster name.

### Runner Def File Schema

`modules/arc-runners/defs/*.yaml` (and `arc-runners-h100/defs`, `arc-runners-b200/defs`):
```yaml
runner:
  name: l-x86iamx-22-225-h100   # ~42 char limit (see docs/runner_naming_convention.md)
  instance_type: p5.48xlarge    # determines node-fleet name (split on ".")
  node_fleet: g5-48xlarge       # optional explicit fleet name; defaults to instance_type.split(".")[0]
  vcpu: 22                      # CAPACITY_AWARE_WORKFLOW_CPU on listener
  memory: 225Gi                 # CAPACITY_AWARE_WORKFLOW_MEMORY
  disk_size: 200                # Gi (per workflow pod)
  gpu: 1                        # 0 for CPU runners
  proactive_capacity: 1         # optional warm pool size (default 0). Reduced fleet-wide in PR #772; most CPU defs now sit at 0-5.
  max_burst_capacity: 250       # optional cap on listener burst (default 0 = unlimited)
  hud_failure_base_capacity: 30 # optional additive floor for HUD-failure fallback (default 0, clamped [0,1000], warn >100). Surge formula: ProactiveCapacity*HUDFailureMultiplier + HUDFailureBaseCapacity. Not capped by clusters.yaml; def value applies in all clusters.
  max_runners: 8                # optional concurrency cap; omit for unlimited (Karpenter scales). May also be a dict: { default: 8, arc-cbr-production-uw1: 48 } — `default` key is mandatory; per-cluster keys override.
  runner_group: default         # GitHub runner group; forced to default for repo-scoped URLs. Cluster-level `arc-runners.runner_group` in clusters.yaml overrides per-def values.
  runner_class: ""              # optional isolation label (e.g. "release")
```

Use `node_fleet` when a runner needs its own dedicated fleet name distinct from the instance family (e.g. to pin the 8-GPU runner to a `-large` companion pool, or to isolate release-class runners). The reserved name `c7i-runner` is rejected — it's the legacy default fleet for plain `c7i.*` runners and cannot be reused as an override (`scripts/python/fleet_naming.py:RESERVED_NODE_FLEET_NAMES`). `derive_fleet_name(instance_type, override)` enforces the DNS-1123 label format on overrides.

Validation: `max_runners` must be a positive int OR a dict containing a `default` key with positive-int values per cluster id (`generate_runners.py:resolve_max_runners`); `max_burst_capacity` must be non-negative; `max_burst_capacity < proactive_capacity` (or `< hud_failure_base_capacity`) is an error, not a warning, and aborts the generate step. `node_fleet`, if present, must be a non-empty string with no leading/trailing whitespace.

**Cluster-wide overrides** (`generate_runners.py`):
- `proactive_capacity_max: <int>` at the cluster level clamps every def's `proactive_capacity` down to that value. Currently only the two staging clusters (`meta-staging-aws-uw1`, `meta-staging-aws-ue1`) set it (to `0`); all prod clusters dropped the override in PR #771 and rely on per-def values reduced in PR #772.
- `pause_runners: true` at the cluster level forces `max_runners=0` and `hud_failure_base_capacity=0` on every scale set — cluster-wide drain switch.
- Region-exclusion auto-zeroing: if a runner's `instance_type` is in the cluster's `excluded_instance_types` (derived from the backing nodepool/fleet's `exclude_regions`, e.g. `us-west-1` for A100/g5), `max_runners`, `proactive_capacity`, and `hud_failure_base_capacity` are forced to 0 so GitHub does not route jobs that would pend forever.
- `arc-runners.runner_group: <name>` at the cluster level pins every scale set's GitHub runner group, unless the cluster's `github_config_url` is repo-scoped — repo-scoped URLs force `default` regardless. Per-cluster groups: `arc-cbr-production-uw1` -> `arc-cbr-prod-uw1`, `meta-prod-aws-ue1` -> `meta-prod-aws-ue1`, `lf-prod-aws-ue1`/`ue2` -> `lf-prod-aws-ue1`/`ue2`, staging -> `meta-staging-aws-uw1`/`ue1`. `arc-cbr-production` (us-east-2) does NOT set `runner_group` and runners land in `default`.

### Cache-Enforcer (DaemonSet)

Uses `xt_string` kernel module to match domain names in TLS ClientHello SNI (port 443) and HTTP Host header (port 80). Rules in CACHE_ENFORCER iptables chain, jumped from OUTPUT and FORWARD. REJECT with tcp-reset (fast "Connection refused").

**Blocked registries** (forced through Harbor at harbor:30002): docker.io, registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com, ghcr.io, nvcr.io, quay.io, registry.k8s.io

**Blocked PyPI** (REJECTed at iptables OUTPUT/FORWARD — the runner pod's pip env vars redirect to pypi-cache instead): pypi.org, files.pythonhosted.org. `download.pytorch.org` is NOT in `PYPI_DOMAINS` — workflows can still reach it directly. Runner pod `PIP_INDEX_URL` / `UV_DEFAULT_INDEX` env vars point at `http://pypi-cache-cpu.pypi-cache.svc.cluster.local:8080/simple/` (Service DNS, not localhost).

**NOT blocked**: public.ecr.aws (no rate limits)

To add/remove blocked domains: edit `REGISTRY_DOMAINS` or `PYPI_DOMAINS` in `modules/cache-enforcer/kubernetes/configmap.yaml`, then `just deploy-module <cluster> cache-enforcer`.

**Limitation**: TLS Encrypted Client Hello (ECH) encrypts SNI, bypassing xt_string matching. No blocked domains currently use ECH. Migration path: Cilium CNI with toFQDNs DNS-based policies.

**Dependency**: cache-enforcer depends on Harbor (base) and pypi-cache module. Without pypi-cache deployed, pip installs fail entirely (traffic blocked but no cache to serve it).

### Node Compactor — Karpenter Integration

Lives at `base/node-compactor/` (NOT under `modules/`). Cluster-level default `node_compactor.enabled: true` in `clusters.yaml`. Per-def override via `node_compactor: true|false` in a NodePool def.

- NodePools labeled `osdc.io/node-compactor: "true"` are auto-discovered
- Compactor taints nodes `NoSchedule` (no eviction of running pods)
- Relies on Karpenter's `WhenEmpty` consolidation policy with `consolidateAfter: 2m` to delete empty tainted nodes
- On SIGTERM, compactor removes all its taints before exiting — redeploying causes temporary burst of untainted nodes
- Burst absorption: when pending pods match tainted nodes (checks tolerations, nodeSelector, affinity, resource fit), compactor temporarily removes taints. Fleet cooldown (default 900s) blocks new taints after burst untaint
- Anti-flap: per-cluster `node_compactor.min_node_age_seconds` (e.g. `arc-cbr-production` sets 900) — newly-launched nodes are exempt from tainting for this many seconds, avoiding a churn loop when Karpenter has just provisioned them for incoming workload

### Runner Pod Two-Tier Resource Model

- **Runner pod** (750m CPU, **1Gi memory**) — lightweight ARC orchestrator; mounts hook ConfigMap. Bumped from 512Mi to 1Gi to give native Node.js stdio buffers (held open by slow CRI exec during pod-density bursts) headroom — observed OOMs traced back to native buffers, not V8 heap. `.NET` runner caps via `DOTNET_GCHeapHardLimit=C800000` (200 MiB), Node hooks via `NODE_OPTIONS=--max-old-space-size=128`. Resources are fixed in `templates/runner.yaml.tpl`, NOT in def files. Listener `CAPACITY_AWARE_RUNNER_CPU/MEMORY` env vars MUST stay in sync with these.
- **Job pod** (resources from def file `vcpu`/`memory`/`disk_size`/`gpu`) — runs actual workflow containers, gets git cache volume.
- Min runners 0; runner scaling is unlimited unless `max_runners` is set in the def. Prefer overspend over outage — only cap fixed-capacity reserved pools (e.g. H100/B200 Capacity Blocks).

**Init container `wait-for-hooks`**: every runner pod runs an Alpine init container that polls `/mnt/host-hooks/dist/index.js` (placed by the `runner-hooks-warmer` DaemonSet in `arc-runners` namespace) for up to 300s, then snapshots `dist/` + `.version` into an emptyDir consumed by the main runner. This is a hard scheduling-gate dependency: nodes without the warmer DaemonSet ready cannot run jobs. Remove when upstream merges the patched hooks.

**Four-tier PriorityClass ladder** (in `modules/arc/kubernetes/priority-classes.yaml`):
- `-10` `placeholder-runner` — proactive runner-capacity placeholders; `preemptionPolicy: Never`
- `0` `arc-runner` — actual runner pods; preempt `placeholder-runner` only
- `10` `placeholder-workflow` — proactive workflow-capacity placeholders; `preemptionPolicy: Never`
- `20` `arc-workflow` — actual workflow (job) pods; preempt `placeholder-workflow`

The pairing ensures workflow pods can claim the capacity reserved by the placeholders they replace, without runner pods clobbering workflow placeholders. Changing any value breaks the proactive-capacity preemption ladder.

**Listener metrics cardinality**: `listenerMetrics` block in the runner template enumerates an allowlist of labels per metric (counters, gauges, histograms) — strips `job_name`, `event_name`, `job_workflow_ref`, `job_workflow_target` which create unbounded series. Keeps `repository`, `organization`, `enterprise`, `job_workflow_name`, `name`, `namespace`, `result`. Required for Grafana Cloud billing predictability; do not add labels without a billing review.

### ARC Module Split

- `modules/arc/` = controller + namespaces (`arc-systems`, `arc-runners`) + runner ServiceAccount + LimitRange + hooks ConfigMap + **`runner-hooks-warmer` DaemonSet** (moved here from `arc-runners` in PR #746 — still runs in the `arc-runners` namespace but lifecycle is now owned by the `arc` module) + four-tier PriorityClass ladder + capacity-monitor RBAC. **No terraform** — pure k8s/helm. Uses fork chart `oci://ghcr.io/jeanschmidt/actions-runner-controller-charts/gha-runner-scale-set-controller`, version pinned in `clusters.yaml` `arc.chart_version` (currently `0.14.1-jeanschmidt.17`). Controller and runner chart versions MUST match (minor mismatch deletes ARS). The cluster-wide containerd registry mirror lives at `base/kubernetes/registry-mirror-config.yaml` — NOT inside the arc module.
- `modules/arc-runners/` = runner scale sets (the actual runners). Separate deploy cycle. Reads `arc-runners.{github_config_url,github_secret_name,runner_name_prefix,runner_group}` from `clusters.yaml`.
- `modules/keda/` = KEDA operator chart (`kedacore/keda`, version pinned via `keda.chart_version`, currently `2.16.1`). Required for BuildKit autoscaling. Deployed on every cluster that enables `buildkit.autoscaling.enabled: true` — currently `meta-staging-aws-uw1`, `meta-staging-aws-ue1`, `arc-cbr-production`, `lf-prod-aws-ue1`, `lf-prod-aws-ue2` (added in PR #775). Does not directly affect runners/nodepools, but the KEDA ScaledObjects in `modules/buildkit/generated/autoscaling.yaml` require this module to be deployed first.

## GPU Nodepools

Three NVIDIA GPU node families are supported, each with a unified single-fleet runner family providing 1/2/4/8-GPU splits via `nvidia.com/gpu` resource requests (NUMA single-numa-node policy).

| GPU | Instance | Module | Capacity model | Notes |
|-----|----------|--------|----------------|-------|
| **A10G** | g5.* (multi-instance fleet: 48xl/12xl/24xl/8xl/16xl, weight-ordered) | `nodepools` | on-demand spot/OD | excludes us-west-1 |
| **L4** / T4 / older | g6, g4dn fleets | `nodepools` | on-demand | shared `nodepools` defs |
| **A100 40GB SXM4** | p4d.24xlarge | `nodepools` (`p4d.yaml` `fleet:` + companion `p4d-large.yaml`) | **on-demand** (no Capacity Block) | excludes us-west-1; packed runners `l-x86iavx512-{11-125-a100, 22-250-a100-2, 44-500-a100-4}` on the `p4d` fleet (single-numa-node), `l-bx86iavx512-88-1000-a100-8` pinned via `node_fleet: p4d-large` (best-effort) |
| **H100 80GB SXM5** | p5.48xlarge | **`nodepools-h100`** (`p5.yaml` + `p5-large.yaml`) + **`arc-runners-h100`** | **AWS Capacity Blocks** (`reserved`; reservation IDs are per-cluster under `clusters.<id>.nodepools-h100.capacity_reservation_ids` — e.g. us-east-2 single `cr-0c3f05dffb85ed832`, us-west-1 dual `cr-04d3d1d84e127a562` + `cr-09a53051589034fb8` for 48 H100s total) | All H100 defs use the per-cluster max_runners dict form. Default values (sized for one 8-GPU node): 1-GPU=8, 2-GPU=4, 4-GPU=2, 8-GPU=1. `arc-cbr-production-uw1` overrides: 1-GPU=48, 2-GPU=24, 4-GPU=12, 8-GPU=6 (sized for the 6 reserved nodes). 8-GPU def pins `node_fleet: p5-large`. Packed `p5.yaml` uses `user_data_script: scripts/h100-node-setup.sh` |
| **B200** | p6-b200.48xlarge | **`nodepools-b200`** (`p6.yaml` + `p6-b200-large.yaml`) + **`arc-runners-b200`** | **AWS Capacity Blocks** (`reserved`, single ID `cr-0b15c0b3163f09d26` in us-east-2) | All B200 defs use plain-int max_runners (no per-cluster dict): 1-GPU=8, 2-GPU=4, 4-GPU=2, 8-GPU=1. 8-GPU def pins `node_fleet: p6-b200-large`. Packed `p6.yaml` uses `user_data_script: scripts/b200-node-setup.sh` |

A100 nodes scale fully with workload (no warm floor — first job after consolidation pays a 5-10 min cold start). H100/B200 nodes are reserved capacity, so concurrency is capped to the reservation.

## Modular Submodule Pattern (GPU pools)

`nodepools-h100`, `nodepools-b200`, `arc-runners-h100`, `arc-runners-b200` are thin shims that **delegate to the base modules** (`nodepools`, `arc-runners`) with overridden defs/output paths. New GPU SKUs can be added as new sibling modules without touching the base.

`modules/nodepools-h100/deploy.sh`:
```bash
UPSTREAM_ROOT="${OSDC_UPSTREAM:-$(cd "$MODULE_DIR/../.." && pwd)}"
export NODEPOOLS_DEFS_DIR="$MODULE_DIR/defs"
export NODEPOOLS_OUTPUT_DIR="$MODULE_DIR/generated"
export NODEPOOLS_MODULE_NAME="nodepools-h100"
exec "$UPSTREAM_ROOT/modules/nodepools/deploy.sh" "$@"
```

`modules/arc-runners-h100/deploy.sh` does the same with `ARC_RUNNERS_DEFS_DIR`, `ARC_RUNNERS_OUTPUT_DIR`, `ARC_RUNNERS_MODULE_NAME`. The base `nodepools/deploy.sh` and `arc-runners/deploy.sh` honor these env vars (with fallback defaults to `$MODULE_DIR/defs`, etc.).

When adding a new GPU SKU, copy one of the `*-h100` modules, swap the def files and reservation ID, register it in `clusters.yaml` `modules:`. Do NOT modify the base `nodepools` or `arc-runners` defs/templates.

### Monitoring Component Placement

- node-exporter tolerates ALL taints (runs on every node)
- DCGM exporter: only on GPU nodes (nodeAffinity on `nvidia.com/gpu.present`)
- All other monitoring components: base infrastructure nodes (tolerate `CriticalAddonsOnly`)

### Karpenter Module = Controller Only

The `modules/karpenter/` module handles the controller only. NodePools are managed by `modules/nodepools/` (and the GPU shims). Karpenter terraform creates: IAM Role (IRSA), SQS Queue (spot/rebalance/health events), four EventBridge Rules (`spot_interruption`, `rebalance`, `instance_state_change`, `scheduled_change` — the last from AWS Health), and discovery tags (`karpenter.sh/discovery`) on cluster SG and private subnets.

---

Verified against commit `c1c884b` (IPv6-only EKS, runner-image per-commit digest pinning via `resolve_runner_version.py`, runner-container-hooks v0.8.15 placed by warmer DaemonSet owned by `arc` module, ARC fork chart `0.14.1-jeanschmidt.16`, keda `2.16.1` enabling BuildKit autoscaling on staging + arc-cbr-production + lf-prod-aws-ue1/ue2, proactive_capacity reductions PR #772 and removal of `proactive_capacity_max` from prod PR #771, integration-test job filtering by cluster module availability PR #774 with cluster-specific runner groups PR #773).