Osdc Project Structure

OSDC Project Structure & Architecture

What This Is

Modular Kubernetes infrastructure platform on AWS EKS. A shared base/ provides the cluster (VPC, EKS, Harbor, GPU plugins, NodeLocal DNS), and optional modules/ layer services on top (ARC, runners, BuildKit, future projects). One codebase drives multiple clusters across regions via clusters.yaml.

Working directory: osdc/. Run all commands from here.

Project Organization — Critical

Before making ANY changes, understand how this project is organized. This project MUST be well-organized at all times:

1. Base vs modules: Shared infrastructure goes in base/. Optional services go in modules/<name>/. If every cluster needs it, it's base.
2. Folder separation is mandatory: Separate directories for terraform/, kubernetes/, helm/, scripts/, docker/ within each component.
3. Technology separation: Don't mix Python, Terraform, Bash, Helm, and Kubernetes manifests in the same directory.
4. Cloud separation: Separate cloud-specific code (AWS/Kubernetes) from cloud-agnostic definitions (runner defs).

Before planning changes:

• FIRST: Examine the current project structure
• THEN: Plan changes to maintain or improve the organization
• NEVER: Mix unrelated files or technologies in the same directory

osdc/
├── clusters.yaml           # THE source of truth — clusters + module lists
├── justfile                # All operations (deploy, lint, test, setup)
├── mise.toml               # Tool versions (tofu, kubectl, helm, crane, ruff, etc.)
├── pyproject.toml          # Python deps + dev deps (managed by uv)
├── scripts/                # Orchestration helpers
│   ├── cluster-config.py   # Reads clusters.yaml, outputs values for just/shell
│   ├── bootstrap-state.sh  # Creates S3 + DynamoDB for tofu state
│   ├── mise-activate.sh    # Sourceable helper for shebang recipes needing mise tools
│   └── python/
│       └── configure_harbor_projects.py  # Harbor proxy cache project setup
├── base/                   # Deployed to EVERY cluster
│   ├── kubernetes/         # StorageClass, NVIDIA plugin, Harbor NS, perf tuning, image-cache-janitor, nodelocaldns, node-taint-remover, registry-mirror config, git-cache, algif-mitigation (CVE-2026-31431, temporary), dirtyfrag-mitigation (CVE-2026-43284 + CVE-2026-43500, temporary)
│   │   ├── eniconfigs/     # AZ-named ENIConfig CRs (one per AZ from private_subnets_by_az terraform output); inert until VPC CNI Custom Networking is enabled. Not used under IPv6-only mode (Custom Networking is IPv4-only); retained as IPv4 rollback safety net
│   │   ├── harbor/         # PDB template (sed-substituted, applied by _deploy-harbor)
│   │   ├── image-cache-janitor/ # Cleans up unused cached container images on nodes
│   │   ├── nodelocaldns/   # NodeLocal DNSCache — per-node CoreDNS DaemonSet (iptables-mode interception); deployed via own deploy.sh (resolves kube-dns ClusterIP at apply time)
│   │   ├── node-taint-remover/ # Operator that strips startup taints once their gating DaemonSets are Ready; ConfigMap registry generated by lib/taint_remover.py, consumed by cache-enforcer, registry-mirror-config, node-performance-tuning
│   │   └── tests/          # Base-level Kubernetes test fixtures
│   ├── helm/               # Harbor values
│   ├── docker/             # Container images (runner-base)
│   ├── scripts/            # Bootstrap (EKS node setup)
│   └── node-compactor/     # Node consolidation controller (taints underutilized nodes)
├── modules/                # Optional, per clusters.yaml
│   ├── eks/                # AWS infrastructure (VPC, EKS, Harbor S3/IAM, image mirroring)
│   ├── karpenter/          # Karpenter controller + AWS infra (IAM, SQS, EventBridge)
│   ├── keda/               # KEDA operator (provides ScaledObject/TriggerAuthentication CRDs used by buildkit autoscaling)
│   ├── arc/                # ARC controller (GitHub Actions)
│   ├── nodepools/          # Karpenter NodePools (pure compute provisioning)
│   ├── nodepools-b200/     # B200 (Blackwell) GPU nodepools
│   ├── nodepools-h100/     # H100 (Hopper) GPU nodepools
│   ├── arc-runners/        # ARC runner scale sets (requires arc + nodepools)
│   ├── arc-runners-b200/   # B200 GPU runner scale sets
│   ├── arc-runners-h100/   # H100 GPU runner scale sets
│   ├── buildkit/           # BuildKit build service (arm64 + amd64, HAProxy LB)
│   ├── cache-enforcer/     # DaemonSet blocking direct registry/PyPI access (forces Harbor + pypi-cache)
│   ├── harbor-cache-recovery/ # Recovers Harbor cache state after pod restarts/failures
│   ├── logging/            # Centralized log collection (Alloy DaemonSet for logs + Deployment for events -> Grafana Cloud Loki)
│   ├── monitoring/         # Metrics pipeline: kube-prometheus-stack CRDs/exporters + Alloy -> Grafana Cloud
│   ├── pypi-cache/         # Self-learning PyPI wheel cache (nginx + pypiserver)
│   └── zombie-cleanup/     # Periodic cleanup of orphaned/zombie pods and resources
├── integration-tests/      # Top-level integration suites (load-test, workflows, workload-test, scripts, docker)
├── tests/smoke/            # Top-level smoke tests
└── docs/                   # Architecture and operations docs

Key Design Decisions

• Harbor is always-on — baked into base, not a module. Every cluster needs a pull-through cache.
• NVIDIA device plugin in base — foundational infrastructure, not optional.
• Karpenter is a module — allows clusters to opt-in to Karpenter. The controller, IAM, SQS, and EventBridge are all in modules/karpenter/. NodePools are separate in modules/nodepools/.
• Compute split: nodepools + arc-runners — nodepools deploys pure Karpenter NodePools (compute provisioning). arc-runners deploys ARC runner scale sets (requires both arc and nodepools). Non-ARC clusters can use nodepools alone for compute.
• One terraform root, many clusters — no code duplication per environment.
• Modules are independent — each has its own terraform state, k8s resources, deploy script. No cross-module imports.
• Monitoring and logging are both modules — metrics collection (Alloy Deployment -> Grafana Cloud Mimir) is opt-in via the monitoring module. Log collection (Alloy DaemonSet for pod/journal logs + Alloy Deployment for Kubernetes events -> Grafana Cloud Loki) is in modules/logging/. Both are secret-gated — no credentials, no Alloy.
• Three Alloy installations — separate Helm releases, namespaces, and RBAC for: monitoring metrics (Deployment, modules/monitoring/helm/alloy-values.yaml), logging pod/journal logs (DaemonSet, modules/logging/helm/alloy-logging-values.yaml), and logging Kubernetes events (Deployment, modules/logging/helm/alloy-events-values.yaml). This avoids config/permission conflicts and lets each scale independently. See osdc-observability skill for details.

External Knowledge Base

The actions-knowledge-base/ directory contains source code, documentation, and detailed reference material for many of the open-source projects and tools used by this project — including Actions Runner Controller, Harbor, the Harbor Helm chart, the GitHub Actions runner, runner images, cloud-provider credential actions, and more. Each project lives as a git submodule under repos/ and the knowledge base's own AGENTS.md indexes every included repository with summaries and key paths.

Finding it: The directory is named actions-knowledge-base and lives somewhere above or beside this project. Walk upward from the current working directory, checking each ancestor and its children, until you find a directory named actions-knowledge-base. It is typically a sibling of the top-level repo (e.g., beside ci-infra/), but the exact location depends on the checkout layout. Do not hardcode a relative path — search for it.

The knowledge base has two key directories:

• actions-knowledge-base/repos/ — Read-only upstream source code and docs (git submodules). Managed via sync.py.
• actions-knowledge-base/docs/ — Our own findings, workarounds, and gotchas discovered during development and operations (e.g., BuildKit OTEL crash, Grafana Alloy setup, CRD ordering issues, deploy phase ordering).

You should consult this knowledge base whenever you need to:

• Understand how a dependency works (e.g., ARC Helm chart values, Harbor configuration, runner lifecycle)
• Clarify configuration quirks, edge cases, or undocumented behavior
• Look up default values, API surfaces, or internal implementation details
• Verify correct usage of upstream Helm charts, container images, or action inputs
• Review operational learnings and previously discovered issues (docs/)

In most cases, reading the relevant source or docs in the knowledge base first will produce more accurate results than guessing or relying on general knowledge alone. Treat it as a first-class resource.

Key Files

Base Component Notes

Node Taint Remover

base/kubernetes/node-taint-remover/ runs an operator that strips startup taints once their gating DaemonSets are Ready on a node. The taint registry is generated by lib/taint_remover.py and consumed by base DaemonSets (cache-enforcer, registry-mirror-config, node-performance-tuning). When adding a new startup-taint-bearing DaemonSet to base or a module, register the taint here.

Node Compactor

base/node-compactor/ is a node consolidation controller that taints underutilized Karpenter nodes to compact workloads WITHOUT eviction. It relies on Karpenter's WhenEmpty consolidation to clean up empty nodes. See osdc-runners-nodepools skill for configuration details.

Documentation (docs/)

Keep this list in sync. When adding, removing, or significantly editing a document in docs/, update this table (description, filename, or entry presence).