# Tf Conventions

> Project-specific Terraform rules for the Azure base stack. Use when writing or reviewing Terraform code in this repo. Trigger when editing *.tf files, reviewing TF diffs, picking an AVM module, deciding on naming, or before terraform apply. Captures the AKS module choice, exact-version pinning policy, naming conventions, API server access posture, NSG hygiene, and the rules that the skills in this repo all follow.

- Skill: `nvidia-omniverse/tf-conventions` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nvidia-omniverse/tf-conventions`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia-omniverse/tf-conventions/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: NVIDIA-Omniverse (https://skillmd.com/u/nvidia-omniverse)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia-omniverse/tf-conventions

---


<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: CC-BY-4.0 AND Apache-2.0 -->

# Terraform conventions for the Azure base stack

These rules apply to any Terraform written in (or generated by) the skills in this repo. They are layered on top of HashiCorp's generic Terraform style guide and the Azure Verified Modules (AVM) requirements — when this skill is silent, defer to those.

## AVM module choice

**Use `Azure/avm-res-containerservice-managedcluster/azurerm` (the resource module).** Not the pattern module (`avm-ptn-aks-production`).

The pattern module hardcodes `private_cluster_enabled = true` and does not expose `api_server_authorized_ip_ranges` or zone selection on additional pools — fine for a fully-private production setup, but blocks the validation flow (kubectl from laptop). The resource module exposes both, plus full control over `default_node_pool`, `network_profile`, `aad_profile`, `api_server_access_profile`, and `oidc_issuer_profile`.

If the project later commits to a fully-private production deploy, re-evaluate the pattern module then.

## Override 1 — exact pin, not pessimistic, for the AVM AKS module

Generic guidance: use `~> #.#` for module versions.

**Project rule:** the AVM AKS module is pinned **exactly**:

```hcl
module "aks" {
  source  = "Azure/avm-res-containerservice-managedcluster/azurerm"
  version = "0.5.4"   # exact, not "~> 0.5"
}
```

**Why:** the module is pre-1.0. A `~>` constraint would auto-pull potentially breaking changes between minor versions. Treat upgrades as code review.

This applies to **the AVM AKS module only.** Provider versions (`azurerm`, `azapi`) follow generic guidance and use `~>`.

## Override 2 — HCL symbolic names vs Azure resource names

HCL symbolic names use lower `snake_case`. The Azure resource `name = "..."` attribute uses **lowercase hyphens**, not snake_case:

```hcl
resource "azurerm_kubernetes_cluster" "main" {     # symbolic name: snake_case
  name                = "aks-nvcf-westus3"         # Azure resource name: hyphens
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
}
```

Naming pattern: `<resource>-<workload>-<env>-<region>` (e.g., `aks-nvcf-validation-westus3`, `rg-nvcf-westus3`).

**Exceptions** — Azure resources with their own naming rules:
- ACR: 5–50 chars, lowercase alphanumeric only, **no hyphens**, globally unique → `acrnvcfwestus3`
- Storage Account: 3–24 chars, lowercase alphanumeric only, **no hyphens**, globally unique → `stnvcfwestus3`
- AKS node pool name: 1–12 chars, lowercase alphanumeric only, **no hyphens** → `gpu`, `compute`

## Project rules

### R1. AVM resource module for the cluster; native resources for additional node pools
Use the AVM resource module pinned `= 0.5.4` for the cluster. Additional `azurerm_kubernetes_cluster_node_pool` resources are fine and expected (needed for explicit zone control on the GPU pool).

### R2. AVM must-haves
- `enable_telemetry = false` (AVM defaults to true)
- `aad_profile = { managed = true, azure_rbac_enabled = true, admin_group_object_ids = [...] }`
- `local_account_disabled = true` — no static admin kubeconfig
- `oidc_issuer_enabled = true` and `workload_identity_enabled = true`
- `api_server_access_profile` with VNet integration + corp-egress allowlist (see R3)
- Pass `principal_id` of `module.aks.kubelet_identity.object_id` to an `azurerm_role_assignment` for `AcrPull` on the ACR; keep ACR in its own module
- Default node pool: `Standard_D8ds_v5` minimum (the AVM default `D4d_v5` is too small once NVCF stack + caches land)

### R3. API server access — VNet integration + corp-egress allowlist (not laptop-IP)

The naive pattern of `enable_private_cluster = false` + `authorized_ip_ranges = [<my-laptop>/32]` is operationally brittle:
- Laptop IPs change frequently (different networks, hotel WiFi, VPN cycling). Every change breaks kubectl.
- In-cluster pods that need to reach the API server's public FQDN (rare but happens) traverse the cluster egress LB → arrive with the LB's public IP as source → fail the allowlist.

Configure as:

```hcl
api_server_access_profile = {
  enable_vnet_integration = true
  subnet_id               = azurerm_subnet.api_server.id  # dedicated /28 in same VNet
  enable_private_cluster  = false                          # keep public FQDN for kubectl/CI
  authorized_ip_ranges    = var.authorized_ip_ranges       # corp-egress CIDR, NOT a /32
}
```

**API server subnet sizing:** minimum `/28` (16 IPs) per AKS docs.

**`authorized_ip_ranges` is corp-egress CIDR, not a laptop /32.** When bringing the cluster up from a network whose egress is off the corp range, VPN to corp first or temporarily widen the allowlist; do not re-pin to the new laptop IP.

Flip to `enable_private_cluster = true` for productionization (no public FQDN; access via jumpbox / Bastion / ExpressRoute). VNet integration also makes that migration cleaner — internal endpoint stays at the same VNet address.

### R4. Cluster autoscaler only
Set `auto_scaling_enabled = true` with `min_count` / `max_count` per pool. No alarm-driven autoscaling layered on top.

### R5. Default to AKS auto-managed NIC NSG; do NOT add a project subnet NSG without explicit reason

**Default:** AKS's auto-managed NIC-level NSG (lives in the `MC_` resource group) is the only Azure-layer ingress filter. The Azure cloud-provider keeps it in sync with every `Service.type=LoadBalancer`. App-port restrictions belong in **K8s NetworkPolicies (Cilium), not the Azure NSG.**

**Do NOT** attach a project-managed `azurerm_subnet_network_security_group_association` to the node subnet for "defense in depth." Two reasons:

1. **Subnet NSG overlays on top of NIC NSG** — both must allow. A subnet NSG that doesn't fully mirror AKS's auto-rules silently breaks Services AKS thought were exposed.
2. **Drift is inevitable** — AKS adds rules dynamically; a static enumeration in TF falls behind.

Add a project-managed subnet NSG only if there is an explicit policy requirement AKS-auto NSG can't express (e.g., restrict ALL inbound to a corp-egress CIDR rather than `Internet`). In those cases the subnet NSG must be a **superset** of AKS-auto rules.

### R6. Cluster TF stack does not own in-cluster k8s objects
The cluster module creates the AKS cluster only. Do not create `kubernetes_storage_class`, `kubernetes_service_account`, `kubernetes_cluster_role*`, `kubernetes_service`, or `kubernetes_config_map` from the cluster module. In-cluster objects belong in helmfile / Argo / a separate platform stack.
**Why:** mixing both into one state file creates destructive ordering bugs and a hard kubeconfig dependency for `terraform plan`.

### R7. AAD + Azure RBAC for Kubernetes
`azure_rbac_enabled = true`. Grant cluster admin via `azurerm_role_assignment` to "Azure Kubernetes Service RBAC Cluster Admin" on the cluster scope. Do not create in-cluster ConfigMaps for admin mapping.

### R8. Workload Identity, not pod-level secrets
`oidc_issuer_enabled = true` and `workload_identity_enabled = true`. Map workload SAs to user-assigned identities via `azurerm_federated_identity_credential`. No connection strings or account keys in K8s Secrets when Workload Identity can do the same thing.

### R9. No `null_resource` + `local-exec`
No waiters, no pollers, no shelling out for image pulls or CIDR overlap checks. Trust the resource graph; use `time_sleep` only when a real eventual-consistency window exists. Image mirroring is operational, not a TF resource.

### R10. No SSH keys generated by Terraform
Don't combine `tls_private_key` + `local_file`. Require a user-supplied `ssh_public_key` variable, or use AAD-based access (`az ssh vm`). Generated keys end up in TF state and on disk.

### R11. Explicit per-env CIDRs in tfvars
Declare `vnet_address_space` and per-subnet CIDRs in tfvars per environment. No algorithmic CIDR allocation, no external scripts to compute or check overlaps.

### R12. Image versions never live in TF
A TF change must never be required to bump an image. Image versions live in the manifest consumed by the mirror script and helmfile — never in `*.tf`.

### R13. State management
- Remote state: `azurerm` backend, separate blob per env.
- Bootstrap the state storage account out-of-band (one-shot script), not in the same TF config that uses it.
- `prevent_destroy` lifecycle on RG, ACR, state storage account, and anything carrying data (Storage Account included).

### R14. Module structure (when promoting from snippets)
- One purpose per module: `iam-vpc`, `aks-cluster`, `aks-nodes`, `acr`, `blob-storage`. Don't bundle.
- Modules accept resource-group name + location as inputs. Don't create RGs inside reusable modules.

### R15. No secrets in `*.tf`, `*.tfvars`, or TF state when avoidable
The NGC API key, AAD admin object IDs, and similar high-sensitivity values come from `streaming-env.sh` (gitignored) or environment variables — never inline in TF, never in committed tfvars.

## Snippet-mode vs structured layout

The skills in this repo emit reviewable artifacts under `generated/`: Stage 02 (`aks-cluster`) emits a Terraform module, while other constructive and validation stages emit runnable scripts unless their own skill states otherwise. These artifacts are working and runnable, but not promoted to a structured `envs/<env>/` layout.

When a deployment graduates from validation to managed lifecycle:
- Promote `generated/<cluster>/aks-cluster/` into `terraform/envs/<env>/aks-cluster/`.
- Apply R13 (remote state, per-env blob).
- All other rules still apply unchanged.

## Pre-merge checklist

Before any `terraform apply` to a non-throwaway env:

- [ ] Every project rule above holds.
- [ ] HashiCorp generic style + AVM MUST-severity rules hold.
- [ ] `terraform fmt -recursive -check -diff` clean.
- [ ] `terraform validate` passes.
- [ ] `terraform plan` reviewed by a human — no surprises.
- [ ] `prevent_destroy` lifecycle on data-bearing resources.
- [ ] No secrets in the diff (`git diff | grep -iE 'nvapi|password|secret|key.*=.*[a-zA-Z0-9]{20,}'`).