Tf Conventions

Terraform conventions for the Azure base stack

These rules apply to any Terraform written in (or generated by) the skills in this repo. They are layered on top of HashiCorp's generic Terraform style guide and the Azure Verified Modules (AVM) requirements — when this skill is silent, defer to those.

AVM module choice

Use Azure/avm-res-containerservice-managedcluster/azurerm (the resource module). Not the pattern module (avm-ptn-aks-production).

The pattern module hardcodes private_cluster_enabled = true and does not expose api_server_authorized_ip_ranges or zone selection on additional pools — fine for a fully-private production setup, but blocks the validation flow (kubectl from laptop). The resource module exposes both, plus full control over default_node_pool, network_profile, aad_profile, api_server_access_profile, and oidc_issuer_profile.

If the project later commits to a fully-private production deploy, re-evaluate the pattern module then.

Override 1 — exact pin, not pessimistic, for the AVM AKS module

Generic guidance: use ~> #.# for module versions.

Project rule: the AVM AKS module is pinned exactly:

module "aks" {
  source  = "Azure/avm-res-containerservice-managedcluster/azurerm"
  version = "0.5.4"   # exact, not "~> 0.5"
}

Why: the module is pre-1.0. A ~> constraint would auto-pull potentially breaking changes between minor versions. Treat upgrades as code review.

This applies to the AVM AKS module only. Provider versions (azurerm, azapi) follow generic guidance and use ~>.

Override 2 — HCL symbolic names vs Azure resource names

HCL symbolic names use lower snake_case. The Azure resource name = "..." attribute uses lowercase hyphens, not snake_case:

resource "azurerm_kubernetes_cluster" "main" {     # symbolic name: snake_case
  name                = "aks-nvcf-westus3"         # Azure resource name: hyphens
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
}

Naming pattern: <resource>-<workload>-<env>-<region> (e.g., aks-nvcf-validation-westus3, rg-nvcf-westus3).

Exceptions — Azure resources with their own naming rules:

• ACR: 5–50 chars, lowercase alphanumeric only, no hyphens, globally unique → acrnvcfwestus3
• Storage Account: 3–24 chars, lowercase alphanumeric only, no hyphens, globally unique → stnvcfwestus3
• AKS node pool name: 1–12 chars, lowercase alphanumeric only, no hyphens → gpu, compute

Project rules

R1. AVM resource module for the cluster; native resources for additional node pools

Use the AVM resource module pinned = 0.5.4 for the cluster. Additional azurerm_kubernetes_cluster_node_pool resources are fine and expected (needed for explicit zone control on the GPU pool).

R2. AVM must-haves

• enable_telemetry = false (AVM defaults to true)
• aad_profile = { managed = true, azure_rbac_enabled = true, admin_group_object_ids = [...] }
• local_account_disabled = true — no static admin kubeconfig
• oidc_issuer_enabled = true and workload_identity_enabled = true
• api_server_access_profile with VNet integration + corp-egress allowlist (see R3)
• Pass principal_id of module.aks.kubelet_identity.object_id to an azurerm_role_assignment for AcrPull on the ACR; keep ACR in its own module
• Default node pool: Standard_D8ds_v5 minimum (the AVM default D4d_v5 is too small once NVCF stack + caches land)

R3. API server access — VNet integration + corp-egress allowlist (not laptop-IP)

The naive pattern of enable_private_cluster = false + authorized_ip_ranges = [<my-laptop>/32] is operationally brittle:

• Laptop IPs change frequently (different networks, hotel WiFi, VPN cycling). Every change breaks kubectl.
• In-cluster pods that need to reach the API server's public FQDN (rare but happens) traverse the cluster egress LB → arrive with the LB's public IP as source → fail the allowlist.

Configure as:

api_server_access_profile = {
  enable_vnet_integration = true
  subnet_id               = azurerm_subnet.api_server.id  # dedicated /28 in same VNet
  enable_private_cluster  = false                          # keep public FQDN for kubectl/CI
  authorized_ip_ranges    = var.authorized_ip_ranges       # corp-egress CIDR, NOT a /32
}

API server subnet sizing: minimum /28 (16 IPs) per AKS docs.

authorized_ip_ranges is corp-egress CIDR, not a laptop /32. When bringing the cluster up from a network whose egress is off the corp range, VPN to corp first or temporarily widen the allowlist; do not re-pin to the new laptop IP.

Flip to enable_private_cluster = true for productionization (no public FQDN; access via jumpbox / Bastion / ExpressRoute). VNet integration also makes that migration cleaner — internal endpoint stays at the same VNet address.

R4. Cluster autoscaler only

Set auto_scaling_enabled = true with min_count / max_count per pool. No alarm-driven autoscaling layered on top.

R5. Default to AKS auto-managed NIC NSG; do NOT add a project subnet NSG without explicit reason

Default: AKS's auto-managed NIC-level NSG (lives in the MC_ resource group) is the only Azure-layer ingress filter. The Azure cloud-provider keeps it in sync with every Service.type=LoadBalancer. App-port restrictions belong in K8s NetworkPolicies (Cilium), not the Azure NSG.

Do NOT attach a project-managed azurerm_subnet_network_security_group_association to the node subnet for "defense in depth." Two reasons:

1. Subnet NSG overlays on top of NIC NSG — both must allow. A subnet NSG that doesn't fully mirror AKS's auto-rules silently breaks Services AKS thought were exposed.
2. Drift is inevitable — AKS adds rules dynamically; a static enumeration in TF falls behind.

Add a project-managed subnet NSG only if there is an explicit policy requirement AKS-auto NSG can't express (e.g., restrict ALL inbound to a corp-egress CIDR rather than Internet). In those cases the subnet NSG must be a superset of AKS-auto rules.

R6. Cluster TF stack does not own in-cluster k8s objects

The cluster module creates the AKS cluster only. Do not create kubernetes_storage_class, kubernetes_service_account, kubernetes_cluster_role*, kubernetes_service, or kubernetes_config_map from the cluster module. In-cluster objects belong in helmfile / Argo / a separate platform stack. Why: mixing both into one state file creates destructive ordering bugs and a hard kubeconfig dependency for terraform plan.

R7. AAD + Azure RBAC for Kubernetes

azure_rbac_enabled = true. Grant cluster admin via azurerm_role_assignment to "Azure Kubernetes Service RBAC Cluster Admin" on the cluster scope. Do not create in-cluster ConfigMaps for admin mapping.

R8. Workload Identity, not pod-level secrets

oidc_issuer_enabled = true and workload_identity_enabled = true. Map workload SAs to user-assigned identities via azurerm_federated_identity_credential. No connection strings or account keys in K8s Secrets when Workload Identity can do the same thing.

R9. No null_resource + local-exec

No waiters, no pollers, no shelling out for image pulls or CIDR overlap checks. Trust the resource graph; use time_sleep only when a real eventual-consistency window exists. Image mirroring is operational, not a TF resource.

R10. No SSH keys generated by Terraform

Don't combine tls_private_key + local_file. Require a user-supplied ssh_public_key variable, or use AAD-based access (az ssh vm). Generated keys end up in TF state and on disk.

R11. Explicit per-env CIDRs in tfvars

Declare vnet_address_space and per-subnet CIDRs in tfvars per environment. No algorithmic CIDR allocation, no external scripts to compute or check overlaps.

R12. Image versions never live in TF

A TF change must never be required to bump an image. Image versions live in the manifest consumed by the mirror script and helmfile — never in *.tf.

R13. State management

• Remote state: azurerm backend, separate blob per env.
• Bootstrap the state storage account out-of-band (one-shot script), not in the same TF config that uses it.
• prevent_destroy lifecycle on RG, ACR, state storage account, and anything carrying data (Storage Account included).

R14. Module structure (when promoting from snippets)

• One purpose per module: iam-vpc, aks-cluster, aks-nodes, acr, blob-storage. Don't bundle.
• Modules accept resource-group name + location as inputs. Don't create RGs inside reusable modules.

R15. No secrets in *.tf, *.tfvars, or TF state when avoidable

The NGC API key, AAD admin object IDs, and similar high-sensitivity values come from streaming-env.sh (gitignored) or environment variables — never inline in TF, never in committed tfvars.

Snippet-mode vs structured layout

The skills in this repo emit reviewable artifacts under generated/: Stage 02 (aks-cluster) emits a Terraform module, while other constructive and validation stages emit runnable scripts unless their own skill states otherwise. These artifacts are working and runnable, but not promoted to a structured envs/<env>/ layout.

When a deployment graduates from validation to managed lifecycle:

• Promote generated/<cluster>/aks-cluster/ into terraform/envs/<env>/aks-cluster/.
• Apply R13 (remote state, per-env blob).
• All other rules still apply unchanged.

Pre-merge checklist

Before any terraform apply to a non-throwaway env:

• Every project rule above holds.
• HashiCorp generic style + AVM MUST-severity rules hold.
• terraform fmt -recursive -check -diff clean.
• terraform validate passes.
• terraform plan reviewed by a human — no surprises.
• prevent_destroy lifecycle on data-bearing resources.
• No secrets in the diff (git diff | grep -iE 'nvapi|password|secret|key.*=.*[a-zA-Z0-9]{20,}').