# Osdc Deployment > OSDC deployment workflow, just recipes, base deploy order, module deploy order, clusters.yaml configuration, Terraform architecture, smoke tests, justfile conventions. Applies to ~/meta/ci-infra/osdc. Load when deploying, adding modules, or modifying deploy scripts. - Skill: `pytorch/osdc-deployment` (Agent Skill) - Install (CLI): `npx skillmds@latest add pytorch/osdc-deployment` - Raw SKILL.md: https://api.skillmd.com/api/skills/pytorch/osdc-deployment/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: pytorch (https://skillmd.com/u/pytorch) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/pytorch/osdc-deployment --- # OSDC Deployment & Operations ## How Deployment Works `clusters.yaml` defines every cluster and its modules. The justfile reads it via `scripts/cluster-config.py`. ```bash just setup # Install Python dependencies (uv sync); mise tools are auto-installed from mise.toml on first `mise exec` just clean # Remove caches, init artifacts, .venv just list # Show all clusters and modules just show # Inspect cluster config just kubeconfig # Configure kubectl (uses scripts/kubeconfig-lock.sh) just bootstrap # Create S3 state bucket + DynamoDB just bootstrap-all # Bootstrap all clusters just deploy # Full deploy (base + modules) just deploy-base # Base only just deploy-module [force] # Single module; force="" sets HELM_FORCE_UPGRADE=1 just plan # Read-only `tofu plan` for base + every TF-backed module (CI-safe) just deploy-history # List deploy audit ConfigMaps in osdc-system namespace just deploy-status [name] # Human-readable status of recent deploys # ^ Both read the deploy-audit ConfigMaps written by every just deploy/deploy-base/deploy-module # into osdc-system. For the ConfigMap schema, raw kubectl queries, cross-cluster version diffs, # stuck-deploy detection, and failure triage, see the "Deploy-Audit ConfigMaps" section of the # osdc-cli-debugging skill. just recycle-nodes # Terminate Karpenter nodes (reprovisioned on demand) just test # Run all unit tests (95% per-file coverage required) just test-compactor # Run node-compactor e2e tests (base/node-compactor/tests/e2e/) just test-janitor # Run image-cache-janitor e2e tests (base/kubernetes/image-cache-janitor/tests/e2e/) just smoke # Run smoke tests against a deployed cluster just integration-test [args...] # Run full integration test via integration-tests/scripts/python/run.py. CI passes --skip-drain --skip-smoke --skip-compactor just generate-arc-runners # Generate ARC scale-set YAMLs (no apply) just lint # Lint all code (shell, Python, Docker, k8s, tf) just lint-fix # Auto-fix lint issues (ruff, shfmt, tofu fmt, taplo) just analyze-utilization # Analyze runner-to-node packing efficiency just simulate-cluster # Monte Carlo cluster simulation for CI load just taint-nodes # Taint ARC runner nodes with NoSchedule for graceful refresh just untaint-nodes # Reverse taint-nodes just drain-runners # Patch all ARC AutoscalingRunnerSets to maxRunners=0, taint runner nodes, wait for in-flight pods to finish (default 1h; tune with OSDC_DRAIN_TIMEOUT_SECS) just resume-runners # Restore maxRunners from defs/ files and untaint nodes (out-of-band recovery; normal cutover path is `just deploy`) just remove-module # Uninstall all osdc.io/module= resources + matching arc-runners helm releases. Run `just drain-runners` first for runner modules. just heal-arc-check [name] # Read-only: detect ARC scale-set-id drift (controller, ARS, listener, ERS). Exit 1 if drift found. just heal-arc [name] # Detect + recover ARC scale-set-id drift (controller restart, ARS re-registration, listener/ERS cascade). just load-test # Run load test against a cluster just workload-test # Run production workload test ``` Internal recipes (prefixed `_`, called from other recipes; documented here because they expose load-bearing behavior): - `_deploy-harbor ` — called from `deploy-base`. Reads `harbor_role_arn`, `harbor_s3_bucket`, `harbor_s3_access_key_id`, `harbor_s3_secret_access_key` from base tofu outputs; installs the Harbor Helm chart; applies the standalone Harbor PDB manifest (chart has no native PDB support); then calls `_configure-harbor-projects`. - `_configure-harbor-projects` — bootstraps Harbor proxy-cache projects via the Harbor REST API. Optionally consumes `harbor-dockerhub-credentials` and `harbor-github-credentials` secrets in `harbor-system` to authenticate proxy-cache pulls against Docker Hub / ghcr.io (avoids anonymous rate limits). Create those secrets out-of-band before deploy if needed. ### Environment variables that control deploy behavior - `OSDC_CONFIRM=yes|no|ask` (default `ask`) — non-interactive confirmation for `just deploy` and `just deploy-base` plans. - `OSDC_TAINT_NODES=yes|no|ask` (default `ask`) — auto-taint ARC runner nodes after `just deploy`. The deploy recipe consults this only when `recycle_karpenter_nodes != true`; under recycle, tainting is skipped unconditionally (nodes are already being destroyed). - `OSDC_SMOKE=yes|no|ask` (default `ask`) — auto-run smoke tests at end of `just deploy`. - `OSDC_DRAIN_TIMEOUT_SECS=` (default `3600`) — wait budget for `just drain-runners` while in-flight runner pods finish before declaring stragglers and exiting non-zero. - `OSDC_UNTAINT_NODES=yes|no` (default `yes`) — controls whether `just resume-runners` removes the refresh taint from runner nodes at the end. Set `no` to leave the taint in place for a separate `just untaint-nodes` step. Mirrors `OSDC_TAINT_NODES` on the `drain-runners` side. - `OSDC_CONFIRM_BASE=destroy` — required *in addition to* `OSDC_CONFIRM=yes` for `scripts/destroy-cluster.sh` to non-interactively destroy base infrastructure. Not consumed by `just deploy`; included here because it sits on the same operational surface. - `OSDC_RESOLVER_READONLY=1` — set by the `smoke` recipe around its `generate-arc-runners` preflight so the runner-image resolver reads the newest cached entry from the lock ConfigMap instead of mutating it or hitting GitHub. Anyone running `generate-arc-runners` standalone for testing should NOT set this unless they explicitly want the cached behavior. - `HELM_FORCE_UPGRADE=1` — passed by `deploy-module` when invoked with the `force` arg; modules' deploy.sh use this to force Helm upgrades. - `AGENT_ENVIRONMENT=true` — agent mode for `just lint`, `just test`, `just smoke`, `just generate-arc-runners`: suppresses output on success, prints only on failure. Useful for sub-agents. - `OSDC_ROOT`, `OSDC_UPSTREAM`, `CLUSTERS_YAML` — exported by `deploy-module`, `deploy-base`, `drain-runners`, `resume-runners`, `smoke`, etc., into the environment of every module `deploy.sh`. `OSDC_ROOT` is the consumer repo root (often equal to `OSDC_UPSTREAM`); `OSDC_UPSTREAM` is the upstream OSDC checkout providing shared scripts and modules. Module scripts that source shared helpers (`mise-activate.sh`, `state-config.sh`, `kubectl-apply.sh`, etc.) MUST use `OSDC_UPSTREAM` so they work both in standalone and consumer-override setups. ### Base deploy order (always) 1. **Terraform** — VPC, EKS, Harbor S3 bucket + IAM role + S3 access keys (all produced by `modules/eks/terraform`, parameterized, no per-env dirs) 2. **Mirror images** — Harbor bootstrap images to ECR (Harbor can't cache itself) 3. **Base k8s** — StorageClass, NVIDIA plugin, performance tuning (`kubectl apply -k`) 4. **node-taint-remover** — Generates the `node-taint-remover-lib` ConfigMap in `kube-system` from `lib/taint_remover.py` (generated at apply time to prevent drift). Mounted into DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning) so they can remove their own startup taint after init. 5. **ENIConfigs** — One AZ-named ENIConfig CR per AZ, rendered from `private_subnets_by_az` (base terraform output). **Inert by design under IPv6-only EKS — VPC CNI Custom Networking is unsupported in IPv6 mode.** Retained on the deploy path as the rollback fallback for IPv4 + Custom Networking. Has its own `deploy.sh` because it must read the terraform output at apply time. 6. **Harbor** — Pull-through cache (caches docker.io, ghcr.io, nvcr.io, registry.k8s.io, quay.io, public.ecr.aws). `_deploy-harbor` reads `harbor_role_arn`, `harbor_s3_bucket`, `harbor_s3_access_key_id`, `harbor_s3_secret_access_key` from base outputs. 7. **Node compactor** — Taints underutilized Karpenter nodes for consolidation (if enabled) 8. **Image Cache Janitor** — Cleans up unused cached images on nodes 9. **NodeLocal DNSCache** — Per-node CoreDNS DaemonSet (iptables-mode interception). Has its own `deploy.sh` (NOT in `base/kubernetes/kustomization.yaml` — same pattern as image-cache-janitor) because it must resolve the live `kube-dns` Service ClusterIP at apply time and substitute it into Corefile + DaemonSet args. ### Module deploy order (per clusters.yaml list order) Each module may have: | File | Purpose | Applied by justfile | |------|---------|-------------------| | `terraform/main.tf` | AWS resources (optional) | `tofu apply` with cluster vars | | `kubernetes/kustomization.yaml` | K8s manifests (optional) | `kubectl apply -k` | | `deploy.sh` | Custom deploy logic (optional) | Called with `(cluster-id, cluster-name, region)` | The justfile auto-detects which of these exist and runs them in order: terraform -> kubernetes -> deploy.sh. `deploy-module` exports three env vars consumed by every module `deploy.sh`: `OSDC_ROOT` (consumer repo), `OSDC_UPSTREAM` (upstream OSDC checkout), and `CLUSTERS_YAML` (path to the active clusters.yaml). It also resolves the module directory with consumer-first precedence: if `$OSDC_ROOT/modules/$MODULE` exists, it wins over `$OSDC_UPSTREAM/modules/$MODULE`. This is how downstream consumers override upstream modules without forking the repo. See `docs/modules.md` for the full module contract. ### clusters.yaml drives everything There is no hardcoded staging/production concept. Each cluster is an independent installation with its own config. `clusters.yaml` controls everything: infrastructure sizing, component tuning (Harbor replicas, Karpenter PDB, ARC log level), module-specific settings (BuildKit replicas, runner GitHub config, max runner counts), and module list. Adding a new cluster: add an entry to `clusters.yaml` with region, cluster_name, state_bucket, base config, component tuning, module config, and module list. Then `just bootstrap ` + `just deploy `. Adding a module to a cluster: append the module name to the cluster's `modules` list. To create a new module: make a directory under `modules/`, add any of the files above, and list the module name in `clusters.yaml` for the target cluster. Reading cluster config in scripts: `uv run scripts/cluster-config.py [default]`. Supports nested paths (e.g., `harbor.core_replicas`). Falls back to `defaults:` section, then optional third argument. **`recycle_karpenter_nodes: true`** (cluster-level config, e.g. `meta-staging-aws-uw1`, `meta-staging-aws-ue1`): when set, `just deploy` calls `just recycle-nodes "$CLUSTER"` automatically after modules deploy and **skips** the ARC node-tainting step (tainting is pointless when nodes are being destroyed). **`proactive_capacity_max: `** (cluster-level): caps the warm-pool placeholder pairs the runner-set scaler will provision ahead of demand. Staging clusters set this to `0` to disable pre-provisioning entirely. See `docs/arc-fork-build-deploy.md` for how this is plumbed into the controller (`CAPACITY_AWARE_PROACTIVE_CAPACITY`). ## Justfile Kubeconfig Convention (MANDATORY) **Every just recipe that interacts with a live cluster MUST call `just kubeconfig ` before doing any kubectl/helm work.** This prevents operators from accidentally running commands against the wrong cluster. The `kubeconfig` recipe wraps `aws eks update-kubeconfig` via `scripts/kubeconfig-lock.sh` — a directory-based mutex (atomic `mkdir`) that serializes concurrent kubeconfig updates and prevents races between parallel deploys. The wrapper enforces a 90s acquisition timeout and breaks stale locks older than 45s. Note: `just plan` does NOT call `just kubeconfig` — it is read-only tofu and does no k8s/helm work, so it is intentionally exempt from the kubeconfig contract. The pattern (in shebang recipes): ```bash export CLUSTERS_YAML="{{CLUSTERS_YAML}}" just kubeconfig "$CLUSTER" ``` This applies to: `deploy-base`, `deploy-module`, `recycle-nodes`, `taint-nodes`, `untaint-nodes`, `drain-runners`, `resume-runners`, `_deploy-harbor`, `smoke`, `test-compactor`, `test-janitor`, `integration-test`, `deploy-history`, `deploy-status`, and any future recipe that calls kubectl, helm, or deploy scripts. When adding a new recipe that touches a cluster, always include the `just kubeconfig` step. Do NOT call `aws eks update-kubeconfig` directly — the locking wrapper is required. ## Justfile + Mise Gotcha Tool versions for the entire repo are pinned in `mise.toml` (tofu, kubectl, helm, crane, awscli, ruff, taplo, shellcheck, shfmt, hadolint, kube-linter, kubeconform, tflint, trivy, uv, python, etc.). The `[settings] auto_install = true` setting means `mise exec` installs missing pinned tools on first use — there is no separate "install tools" command; `just setup` only runs `uv sync` for Python deps. The justfile uses `set shell := ["mise", "exec", "--", "bash", "-euo", "pipefail", "-c"]` so that mise-managed tools are on PATH for every recipe line. **Shebang recipes bypass this.** A recipe starting with `#!/usr/bin/env bash` is executed as a standalone script by just, NOT through the configured shell. This means mise-managed tools may not be on PATH inside shebang recipes or scripts they call. Shebang recipes in this repo work around it by sourcing `scripts/mise-activate.sh` as their first line. For recipes that need mise tools, use non-shebang style (line-by-line with `@` prefix) so they run through `mise exec`. If a shebang recipe is required, source `mise-activate.sh` (or call subscripts via `mise exec -- ./script.sh` explicitly). ## Terraform Architecture Single parameterized root at `modules/eks/terraform/`. No per-environment directories. - Variables (`cluster_name`, `aws_region`, `vpc_cidr`, etc.) come from `clusters.yaml` -> justfile -> `tofu -var=` - Backend configured at `tofu init` via `-backend-config` (bucket, key, region) - Each cluster gets its own state: `s3:////base/terraform.tfstate` - Modules with their own terraform get: `s3://///terraform.tfstate`. Module terraform also receives `state_bucket` and `cluster_id` as vars. ## Smoke Tests (MANDATORY) Post-deploy validation that a cluster is actually healthy. Run via `just smoke ` or automatically at the end of `just deploy` (controlled by `OSDC_SMOKE=yes|no|ask`). Smoke tests verify that what was deployed is actually working: Helm releases are deployed, pods are Running, DaemonSets have all pods ready, required secrets and service accounts exist, AWS resources (IAM roles, SQS queues) are present, and Karpenter NodePools / ARC runner scale sets match their definition files. They do NOT test application-level behavior — just that the deployment landed correctly. **Architecture**: Each module owns its own smoke tests, co-located at `/tests/smoke/`. A root `tests/smoke/conftest.py` provides shared fixtures (batch-fetched cluster state, cluster config, CLI helpers) that all module tests inherit via pytest's conftest discovery. `just smoke ` reads `clusters.yaml`, finds enabled modules, collects their `tests/smoke/` directories, and runs pytest across all of them. The recipe also unconditionally pulls smoke tests from `base/**/tests/smoke` AND `modules/eks/tests/smoke` (EKS is always deployed as part of base, regardless of `clusters.yaml.modules`). Adding a module with smoke tests = they're discovered automatically. Moving a module = its tests move with it. **Preflight: ARC runner YAML pre-generation.** Before any pytest worker starts, the `smoke` recipe iterates the cluster's enabled modules and runs `just generate-arc-runners "$CLUSTER"` for every `arc-runners` / `arc-runners-*` variant (canonical + per-GPU-arch), parameterized via `ARC_RUNNERS_DEFS_DIR` / `ARC_RUNNERS_OUTPUT_DIR` / `ARC_RUNNERS_MODULE_NAME`. This regeneration MUST happen once up front — previously a test fixture shelled out per-test and raced under pytest-xdist. Anyone debugging "why does smoke regenerate before tests run" is looking at this step. **When changing ANY module or base component, you MUST update its smoke tests to cover the change.** If the component doesn't have smoke tests yet, add them. Smoke tests are the safety net that catches deployment regressions — they only work if they stay current. - Smoke test files live in `/tests/smoke/test_*.py` - Shared helpers live in the `tests/smoke/helpers/` package: - `helpers/cli.py` — `run_kubectl`, `run_helm`, `run_aws` (proxy-bypass-aware) - `helpers/filters.py` — pure in-memory filters over batch fixtures - `helpers/nodes.py` — node-churn helpers (unstable nodes, disruption taints) - `helpers/retry.py` — `retry_with_backoff` primitive - `helpers/k8s_asserts.py` — DaemonSet / Deployment readiness assertions - `helpers/remote.py` — Mimir / Loki query helpers with skip-on-network-failure semantics - `helpers/__init__.py` re-exports everything (`from helpers import X` works without naming the submodule) - Each module's `tests/smoke/conftest.py` re-exports the root conftest: `from smoke_conftest import *` - DaemonSet checks use `assert_daemonset_healthy()` which tolerates mismatches caused by nodes in transition (new, NotReady, or being deleted). Deployment checks use `assert_deployment_ready()` which has a 90-second rollout-completion window followed by the standard `retry_with_backoff` schedule once the rollout finishes but replicas still aren't ready - **Retry primitive**: `retry_with_backoff(check, refresh=None, delays=None)` is the canonical retry abstraction for smoke tests. Retries only on `AssertionError` — other exceptions propagate immediately. Optional `refresh` callable runs between attempts to re-fetch live state. Re-raises the LAST `AssertionError` after exhaustion. - **Backoff schedule** (resolved at module load time from `os.environ.get("CI")`): - **Local** (default): 6 attempts total — 1 initial + 5 retries. Sleeps `[10, 17, 28.9, 49.1, 83.4]` seconds. Worst-case sleep budget per assertion: ~188s. - **CI** (`CI=true`, set by Buildkite/GHA/etc.): 8 attempts total — extended to `[10, 17, 28.9, 49.1, 83.4, 141.78, 241.03]` to give slow recoveries more time. Worst-case sleep budget: ~571s per assertion. - **When writing new smoke tests**: use `retry_with_backoff` for any "wait for X to be ready" pattern — do NOT hand-roll `for _ in range(N): time.sleep(D)` loops. ## Unit Tests & Coverage `just test` discovers `test_*.py` files across the repo, runs them with pytest-xdist (`-n auto`), and enforces a **per-file coverage threshold of 95%**. Any source file under the test dirs whose coverage drops below 95% will fail the run, even if pytest itself passed. Add or extend tests when you change source files — bumping global coverage is not enough. Exclusions and gotcha around smoke tests: - Excluded entirely: `tests/e2e/`, `*/base/*/tests/smoke/`, `*/modules/*/tests/smoke/` (live-cluster suites — run via `just smoke `). - **Included**: the project-root `tests/smoke/` directory, because it hosts unit tests for the smoke-helpers package itself (e.g. `tests/smoke/helpers/test_retry.py`). The recipe runs those tests but **exempts them from the 95% per-file coverage gate** (the helpers' full coverage comes from live-cluster smoke runs). ## Module-Specific Deployment Notes ### EKS Terraform State Key Gotcha The eks module uses `/base/terraform.tfstate` as the state key (historical, predates module extraction). Don't assume the state key matches the module name `eks`. Modules reading eks outputs via `terraform_remote_state` (e.g., karpenter reads OIDC provider, node role, SG, subnets) depend on this key. ### Base Node Group — Hidden Behaviors The `aws_eks_node_group "base"` resource at `osdc/modules/eks/terraform/modules/eks/main.tf` has two non-obvious behaviors that matter when changing `base_node_count` or `base_node_instance_type` in `clusters.yaml`. #### `base_node_count` is half-applied by tofu `lifecycle.ignore_changes = [scaling_config[0].desired_size]` means tofu updates `min` and `max` but NOT `desired`. After bumping `base_node_count`, you MUST manually run: ``` NO_PROXY="$NO_PROXY,.eks.amazonaws.com" no_proxy="$no_proxy,.eks.amazonaws.com" \ aws eks update-nodegroup-config \ --cluster-name \ --nodegroup-name -base-nodes \ --region \ --scaling-config minSize=N,maxSize=N,desiredSize=N ``` Skipping this leaves the ASG at the OLD `desired_size`. Symptom: pods go Pending despite the new count being "applied". #### `base_node_instance_type` change triggers an in-place rolling replacement EKS treats an `instance_types` change as a managed node group update: every node is cordoned, drained, and replaced. - Honors PDBs for 15 minutes (the `update_config` timeout) - Then forces eviction because `force_update_version = true` - Blast radius controlled by `base_node_max_unavailable_percentage` in `clusters.yaml` (default 33%, staging 100%) Monitor the rolling update with: ``` aws eks list-updates --cluster-name --nodegroup-name -base-nodes --region aws eks describe-update --cluster-name --nodegroup-name -base-nodes --region --update-id ``` For one-time migrations (e.g. switching to a larger instance family), follow the cluster-specific playbook if one exists at the repo root — disruption coordination, manual `desired_size` bump, and rollback steps live there, not here. If no playbook exists for your migration, write one before starting. **Cluster recreation runbook**: - `docs/cluster-recreation.md` — operator-facing runbook for destroying and recreating an existing OSDC cluster (used for any change that touches an immutable `aws_eks_cluster` property: IP family, VPC, CIDR, encryption_config, etc.). Cluster recreates are required because EKS does not support in-place IPv4↔IPv6 conversion. Automation: `scripts/destroy-cluster.sh ` (requires `OSDC_CONFIRM=yes` + `OSDC_CONFIRM_BASE=destroy` for non-interactive destroy). ### CRD Ordering — Monitoring Module Monitoring deploy order is strict: (1) namespace + DCGM DaemonSet via kustomization, (2) kube-prometheus-stack Helm (installs CRDs + exporters), (3) `monitors/` applied AFTER CRDs exist, (4) Alloy conditionally. Applying ServiceMonitor/PodMonitor resources before kube-prometheus-stack installs the CRDs will fail. ### Cache-Enforcer Dependency cache-enforcer depends on Harbor (base) and pypi-cache module. Without pypi-cache deployed, pip installs in runner jobs fail entirely (traffic blocked but no cache). Deploy pypi-cache before or alongside cache-enforcer. ### clusters.yaml Config Keys Reference `clusters.yaml` itself is the source of truth — the section below is a curated subset highlighting load-bearing or non-obvious keys. For the full schema (every `defaults:` and every per-cluster override), read `clusters.yaml` directly. **Cluster-level keys** (set under `clusters.:`, no fallback to `defaults:`): - `region` — AWS region for the cluster (e.g. `us-west-2`). - `cluster_name` — EKS cluster name (used in IAM roles, ECR repo names, kubeconfig alias). - `state_bucket` — S3 bucket holding terraform state for THIS cluster (`s3:////...`). - `recycle_karpenter_nodes: true|false` — when true, `just deploy` runs `just recycle-nodes` after modules and skips ARC taint-nodes. - `access_config:` — EKS authentication mode + cluster-admin access entries: ```yaml access_config: authentication_mode: API_AND_CONFIG_MAP cluster_admin_role_names: - osdc_gha_prod ``` Consumed by `scripts/cluster-config.py` into tofu vars. Without this, no IAM role gets cluster-admin via access entries. **Karpenter:** ```yaml karpenter: replicas: 4 # Controller replica count log_level: info pdb_enabled: true pdb_min_available: 1 ``` **Harbor:** ```yaml harbor: core_replicas: 4 registry_replicas: 4 nginx_replicas: 6 pdb_max_unavailable: 1 # Default: conservative production protection for core/registry/nginx # staging clusters (meta-staging-aws-uw1, meta-staging-aws-ue1) override to "100%" (string, quoted) # to effectively disable — staging runs 1 replica each ``` Standalone PDB manifests are applied by `_deploy-harbor` after the Helm install (Harbor chart has no native PDB support). See `osdc-harbor` skill for selector details and known limits. **ARC controller:** ```yaml arc: chart_version: "0.14.1-jeanschmidt.17" # Must match controller + runner charts (minor mismatch deletes ARS). Fork chart from jeanschmidt/actions-runner-controller. Check clusters.yaml for the current value. runner_image_tag: "2.334.0" replica_count: 4 log_level: info controller_cpu_request: "1" controller_cpu_limit: "4" controller_memory_request: "2Gi" controller_memory_limit: "4Gi" ``` **BuildKit:** Defaults provide `replicas_per_arch`, both instance-type maps, and `autoscaling.enabled`. Every cluster inherits the instance-type maps; the remaining knobs are per-cluster overrides: ```yaml defaults: buildkit: replicas_per_arch: 12 # Used when a cluster omits per-arch replica counts amd64_instance_types: # instance type -> pods per node, in preference order m6id.24xlarge: 2 m6id.12xlarge: 1 arm64_instance_types: m7gd.16xlarge: 4 autoscaling: enabled: false clusters: my-cluster: buildkit: amd64_replicas: 32 arm64_replicas: 8 ``` Per-arch override keys are separate (`amd64_*` and `arm64_*`) — there is no flat `pods_per_node` key. `{amd64,arm64}_instance_types` maps each instance type to its pods per node, in preference order. Each entry becomes its own weighted NodePool, so the first is what a normal scale-up uses and later ones are only reached when the one above cannot provision (ICE, or the pool's limits). One Deployment means one pod spec: the first entry sizes the pod, and a later entry must hold at least its stated count of that pod or generation fails. No cluster currently overrides the maps — override only to give one cluster a different family or a different preference order. **Node Compactor:** All config under `node_compactor:` key. Knobs configurable via `clusters.yaml`: `enabled`, `interval_seconds` (20), `dry_run`, `min_nodes` (1), `min_node_age_seconds` (900), `capacity_reservation_nodes` (0), `max_uptime_hours` (48). Other knobs (`taint_rate`, `fleet_cooldown`, `spare_capacity_nodes`, `spare_capacity_ratio`, `spare_capacity_threshold`) are internal Python defaults in the compactor source, NOT configurable via clusters.yaml. **Other defaults blocks** (curated — see `clusters.yaml` `defaults:` for the full list and current values): - `coredns.replicas` — CoreDNS replica count. - `nodepools.{gpu_consolidate_after,cpu_consolidate_after,baremetal_consolidate_after,gpu_disruption_budget,cpu_disruption_budget}` — Karpenter NodePool churn knobs. - `monitoring.{grafana_cloud_url,grafana_cloud_read_url}` — Grafana Cloud Mimir push/read endpoints. - `logging.grafana_cloud_loki_url` — Grafana Cloud Loki push endpoint. - `alloy_chart_version` — Alloy Helm chart version (shared by monitoring + logging Alloy DaemonSets). - `keda.chart_version` — KEDA operator Helm chart version (used by the `keda` module; consumed by buildkit autoscaling). - `zombie_cleanup.{enabled,pending_max_age_hours,running_max_age_hours,busy_max_age_hours,dry_run}` — orphan-pod GC tuning. - `pypi_cache.{instance_type,cuda_versions,python_versions,target_architectures,target_manylinux}` — PyPI wheel cache matrix (see `osdc-pypi-cache` for slug semantics; `cuda_versions` and `python_versions` are mandatory for this module). ### ARC Module — No Terraform Unlike most modules, `modules/arc/` has no terraform phase — it is pure k8s/helm. Don't look for terraform state or try `just tf-*` recipes for ARC. ### Pypi-Cache — Two Terraform Roots The pypi-cache module has TWO terraform roots; only one runs as part of `just deploy`: - `modules/pypi-cache/terraform/main.tf` — per-cluster IRSA role, IAM policies, etc. **Auto-applied by `just deploy-module pypi-cache`** (the standard `deploy-module` flow detects `terraform/main.tf` and runs `tofu init && tofu apply` on it). - `modules/pypi-cache/terraform/wheel-cache-bucket/` — standalone tofu root for the shared S3 bucket (`pytorch-pypi-wheel-cache`). This is a one-time, account-wide resource, **NOT part of `just deploy`**. Run `tofu init && tofu apply` manually in this sub-directory if the bucket needs to be created or updated. ### Pypi-Cache — Required clusters.yaml Keys `cuda_versions` and `python_versions` are NOT module defaults. They must be configured in `clusters.yaml` under `defaults.pypi_cache` or per-cluster `clusters..pypi_cache`. Omitting them produces zero CUDA slugs and zero deployments. `cuda_slug()` normalizes versions to major.minor only: `"12.8.1"` → `cu128` (matches PyTorch convention). ## CI Workflows OSDC is driven from CI via GitHub Actions workflows living in the **parent repo** (`ci-infra/.github/workflows/`), NOT inside `osdc/`. There is no `osdc/.github/` directory. Key workflows (all callable as reusable workflows via `workflow_call` from the orgs that ship from this repo): | Workflow | What it runs | |----------|-------------| | `osdc-lint.yml` | `just lint` | | `osdc-test.yml` | `just test` | | `osdc-pre-merge.yml` | Lint + test on PR | | `osdc-plan-prod.yml` | `just plan ` (read-only tofu plan; CI-safe) | | `osdc-deploy-prod.yml` | Per-cluster prod deploy. Calls `_osdc-deploy.yml` | | `_osdc-deploy.yml` | Reusable: `just lint && just test` → `just deploy ` → `just smoke ` → `just integration-test --skip-drain --skip-smoke --skip-compactor`. Pre-flight checks can be skipped with the `skip_lint_test` input (firefighting only). | | `_osdc-plan.yml` | Reusable: `just plan ` and tee to `plan.txt` | | `_osdc-slow-tests.yml` | Reusable: `just load-test`, `just test-compactor`, `just test-janitor` | | `osdc-capacity-report.yml` | `just simulate-cluster` + `just analyze-utilization`, periodic | When adding a new `just` recipe that CI should run, plumb it through the corresponding reusable workflow. The OSDC-rooted CI jobs — the `deploy`/`smoke`/`integration` jobs in `_osdc-deploy.yml`, plus `osdc-drain.yml` and `osdc-undrain.yml` — set up their toolchain via the local composite action `./.github/actions/osdc-setup` (installs just + mise + `uv sync`). Its setup steps are retried on transient failures by **retry-by-duplication** (`continue-on-error` on the first attempt plus a copy gated on `steps..outcome == 'failure'`), chosen over a retry action deliberately: GitHub Actions has no native step retry, and a retry action would run unpinnable third-party code on the deploy / merge-queue path. `actions/checkout` and `aws-actions/configure-aws-credentials` are intentionally NOT wrapped because they already retry internally. On the deploy path, the `smoke` step is also retried once by duplication, with a 180s settle wait before the retry, because smoke failures are usually transient post-deploy cluster churn.