Iac

SPI Infrastructure as Code

Terraform, Helm, and Azure PaaS infrastructure for OSDU SPI, with systematic debugging and evidence-based verification.

Quick Start

Before first use, verify tools are available:

terraform --version && helm version --short && kubectl version --client && az version && azd version

If any command is not found, stop and use the setup skill to install missing dependencies.

Project Architecture

Three-layer Terraform architecture with independent state per layer, orchestrated by azd:

osdu-spi-infra/
├── azure.yaml                  # azd project definition
├── infra/                      # Layer 1: AKS + Azure PaaS (~15 min)
│   ├── aks.tf                  # AVM AKS Automatic module
│   ├── cosmosdb.tf             # Gremlin (entitlements) + SQL (per partition)
│   ├── servicebus.tf           # Per-partition topics
│   ├── storage.tf              # Common + per-partition accounts
│   ├── keyvault.tf             # Secrets and certificates
│   ├── identity.tf             # Workload Identity + federated credentials
│   ├── monitoring.tf           # App Insights, Log Analytics, Prometheus
│   ├── locals.tf               # 24 CosmosDB containers, 14 Service Bus topics
│   └── outputs.tf              # 40+ outputs for downstream layers
├── infra-access/               # Layer 1a: RBAC bootstrap (~1 min)
│   └── main.tf                 # Elevated role assignments
├── software/
│   ├── foundation/             # Layer 2: Cluster operators (~3 min)
│   │   ├── main.tf             # cert-manager, ECK, CNPG, ExternalDNS, Gateway
│   │   └── charts/             # Helm charts per operator
│   └── stack/                  # Layer 3: Middleware + OSDU services (~5 min)
│       ├── middleware.tf        # Elasticsearch, PostgreSQL, Redis, Airflow
│       ├── osdu-services-core.tf      # 11 core services
│       ├── osdu-services-reference.tf # 3 reference services (CRS, Unit)
│       ├── platform.tf         # Namespaces, Karpenter NodePool, Istio mTLS
│       └── modules/            # Helm-based components
│           └── osdu-service/   # Reusable OSDU service wrapper
└── scripts/                    # azd lifecycle hooks (PowerShell 7.4+)

Key separation:

• infra/ — Azure resources (AKS, CosmosDB, Service Bus, Storage). State managed by azd.
• infra-access/ — Elevated RBAC. Separate state for privilege isolation.
• software/foundation/ — Cluster-wide operators. Independent Terraform state.
• software/stack/ — OSDU services and middleware. Independent Terraform state.
• Cross-layer values flow through azd environment variables, never direct state references.

Azure PaaS Resources

CosmosDB

Two types per deployment:

Gremlin (1 account) — Entitlements graph database:

resource "azurerm_cosmosdb_account" "gremlin" {
  kind = "GlobalDocumentDB"
  capabilities { name = "EnableGremlin" }
}

SQL (1 per partition) — Operational data with 24 containers:

resource "azurerm_cosmosdb_account" "sql" {
  for_each = toset(var.data_partitions)
  kind     = "GlobalDocumentDB"
}

Containers include: Authority, EnvConfigValues, FileLocationEntity, IngestionStrategy, LegalTag, MappingInfo, RegisterAction, RegisterDdms, RegisterSubscription, Schema, SchemaInfoRepository, StorageRecord, StorageSchema, Tags, TenantInfo, WorkflowCustomOperatorInfo, WorkflowRunStore, WorkflowV2, and more.

Service Bus

Per-partition namespaces with 14 topics:

indexing-progress, legaltags-changed, records-changed, schema-changed,
storage-records-changed, storage-records-deleted, file-generated,
csv-parser-status, status-changed, topic-wks, reindex-topic,
gcm-topic, replay-topic, search-event

Azure Storage

• Common account — System data, Airflow DAGs, CRS conversion data
• Per-partition accounts — Legal configs, file area, WKS mappings

Key Vault

Single vault storing connection strings, credentials, and certificates. Accessed via Workload Identity — no long-lived secrets in pods.

Multi-Partition Support

variable "data_partitions" {
  type    = list(string)
  default = ["opendes"]
}

Each partition gets its own CosmosDB SQL account, Service Bus namespace, and Storage account. System database is created only on the primary partition (first in list).

Use for_each = toset(var.data_partitions) for partition-scoped resources. Use indexing (var.data_partitions[0]) only for the primary partition.

Workload Identity

Azure AD federated credentials across 8 namespaces: default, osdu-core, airflow, osdu-system, osdu-auth, osdu-reference, osdu, platform

Each namespace gets a ServiceAccount with the annotation:

azure.workload.identity/client-id: <managed-identity-client-id>

No CSI secret store driver needed — pods authenticate directly to Azure PaaS via federated tokens.

Feature Flags

Per-service enable/disable in stack variables:

# Core services
variable "enable_partition"    { default = true }
variable "enable_entitlements" { default = true }
variable "enable_legal"        { default = true }
# ... etc for all 11 core + 3 reference services

# Middleware
variable "enable_elasticsearch" { default = true }
variable "enable_airflow"       { default = true }
variable "enable_redis"         { default = true }

# Networking
variable "enable_gateway"           { default = true }
variable "enable_osdu_api_ingress"  { default = true }
variable "enable_external_dns"      { default = true }

Use count patterns: count = var.enable_partition ? 1 : 0

Blue/Green Stacks

The STACK_NAME variable enables parallel deployments on the same cluster:

# Default stack
azd env set STACK_NAME ""          # Namespaces: platform, osdu

# Blue stack
azd env set STACK_NAME "blue"      # Namespaces: platform-blue, osdu-blue

Useful for zero-downtime upgrades and canary deployments.

azd Lifecycle Hooks

PowerShell 7.4+ scripts in scripts/:

Important: These are PowerShell scripts (.ps1), not bash. Debugging requires pwsh to be installed. See the setup skill.

Postrender + Kustomize for AKS Safeguards

Charts that don't expose safeguard fields need postrender:

resource "helm_release" "operator" {
  name  = "my-operator"
  chart = "my-operator"
  postrender {
    binary_path = "${path.module}/kustomize/postrender.ps1"
  }
}

Note: SPI uses PowerShell postrender scripts (postrender.ps1), not bash.

AKS Automatic Deployment Safeguards are the same as CIMPL:

• All containers: readinessProbe, livenessProbe, resource requests, seccompProfile: RuntimeDefault
• Replicas > 1: topologySpreadConstraints or podAntiAffinity
• Forbidden: :latest tags, privileged containers, NET_ADMIN/NET_RAW

Key Difference from CIMPL

Both use AKS Automatic, managed Istio, cimpl-helm charts, and azd orchestration.

Debugging Infrastructure Issues

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Four-Phase Process

Phase 1: Gather Evidence

STOP. Do not attempt any fix yet.

# Terraform state
terraform -chdir=infra state list | head -30
terraform -chdir=software/stack state list | head -30

# Azure PaaS health
az cosmosdb show --name <name> -g <rg> --query provisioningState
az servicebus namespace show --name <name> -g <rg> --query status
az storage account show --name <name> -g <rg> --query statusOfPrimary

# Cluster health
kubectl get pods -A --field-selector status.phase!=Running
kubectl get events --sort-by=.lastTimestamp -A | tail -20

# azd state
azd env get-values

Phase 2: Recent Changes

git log --oneline -10
terraform -chdir=infra state pull | jq '.serial'

Phase 3: Form Hypothesis

State a single, testable theory: "The Service Bus topic X is not receiving messages because the Workload Identity federation is missing for namespace Y."

Phase 4: Verify Fix

Make the minimal change. Verify with fresh evidence. Document what was wrong and why.

NO COMPLETION CLAIMS WITHOUT FRESH VERIFICATION EVIDENCE