Infrastructure As Code

Infrastructure As Code

When to Use

Use this skill when:

• The user asks about writing, structuring, or managing Terraform, Pulumi, AWS CDK, Bicep, Crossplane, or Ansible configurations for production infrastructure
• The user wants to migrate from manual ClickOps provisioning or shell scripts to declarative IaC
• The user needs to design module structure, state management strategy, or workspace layout for a multi-environment IaC project
• The user asks about drift detection, plan validation, policy-as-code (OPA, Sentinel, Checkov), or IaC testing pipelines
• The user needs to implement GitOps-style infrastructure delivery with PR-based plan review and apply gates
• The user is debugging a Terraform state conflict, import block, or provider version mismatch
• The user wants to refactor a monolithic Terraform root module into composable, versioned modules
• The user asks about secret injection, remote state backends, or workload identity for IaC pipelines

Do NOT use this skill when:

• The user needs container orchestration specifics (Kubernetes manifests, Helm charts, Kustomize) -- use the container-orchestration skill instead
• The user is asking about CI/CD pipeline configuration (GitHub Actions YAML, Tekton pipelines) for application code rather than infrastructure -- use the ci-cd-pipelines skill
• The user needs cloud cost optimization analysis without IaC context -- use the cloud-cost-management skill
• The user wants operational runbooks or incident response for live infrastructure failures without an IaC angle
• The user is asking about application configuration management (environment variables, feature flags) not infrastructure provisioning
• The user needs networking design theory without IaC implementation -- use the cloud-networking skill

Process

1. Identify the IaC Toolchain and Scope

Before writing a single line of HCL or TypeScript, establish the exact context:

• Tool selection criteria: Use Terraform/OpenTofu when the team spans multiple cloud providers or already has HCL expertise. Use AWS CDK (TypeScript) when the team is AWS-native and prefers imperative constructs and strong typing. Use Pulumi when the team wants general-purpose languages (Python, Go, TypeScript) and avoids DSL learning overhead. Use Bicep/ARM when the organization is Microsoft-first and needs tight Azure policy integration. Use Crossplane when the control plane must run inside Kubernetes and platform teams own cloud resource APIs.
• Scope boundaries: Determine what is in scope for IaC vs. what is managed by other systems. Kubernetes workloads, DNS records managed by external-dns, and database schemas managed by Flyway are typically out of scope for Terraform.
• State of existing infrastructure: If resources already exist, plan the import strategy before writing any new code. Terraform 1.5+ import blocks allow declarative, PR-reviewable imports. For large existing estates, use Terraformer or aws2tf to generate a baseline, then clean up generated code before committing.
• Team scale signals: 1-3 engineers -- monorepo with a single state backend and simple workspace separation is fine. 4-15 engineers -- split by functional domain (networking, compute, data) with separate state files and CI-gated applies. 15+ engineers -- full platform team model with published module registry and self-service via Terraform Cloud or Atlantis.

2. Design the Repository and Module Structure

Module structure decisions made at the start compound in either direction over years:

• Root module vs. child module distinction: Root modules are the entry points that get planned and applied (they have terraform.tfvars, backend.tf, and environment-specific values). Child modules are reusable units that accept variables and produce outputs -- they should never have backend configuration.
• Recommended monorepo layout for Terraform:

  infra/
  ├── modules/
  │   ├── networking/        # VPC, subnets, NAT gateways
  │   ├── eks-cluster/       # EKS control plane + node groups
  │   ├── rds-postgres/      # RDS instance + parameter groups + secrets
  │   └── observability/     # CloudWatch, Grafana, Prometheus stack
  ├── environments/
  │   ├── dev/
  │   │   ├── main.tf        # Calls modules with dev-appropriate values
  │   │   ├── terraform.tfvars
  │   │   └── backend.tf
  │   ├── staging/
  │   └── prod/
  └── .github/
      └── workflows/
          └── terraform.yml
  

• Module versioning: Once a module is consumed by multiple environments or teams, version it via Git tags (v1.2.0) and reference it with source = "git::https://github.com/org/infra-modules.git//eks-cluster?ref=v1.2.0". Never pin to a branch in production. Use a private Terraform Registry (Terraform Cloud, JFrog) for enterprise module distribution.
• Separation of state files by blast radius: Networking state should be separate from compute state. A plan/apply failure in the compute layer must never lock or corrupt the networking state. Group resources by: (a) how often they change, and (b) what the blast radius is if apply fails.
• Naming conventions: Every resource name must encode: environment, region, team/service, resource type. Example: prod-us-east-1-payments-rds-pg. This prevents name collisions and makes resource ownership unambiguous in cost reports.

3. Configure Remote State and Locking

Unmanaged state is the single biggest source of IaC incidents:

• Backend selection: Use S3 + DynamoDB for AWS-native teams (DynamoDB provides optimistic locking; use PAY_PER_REQUEST billing). Use Azure Blob Storage with lease-based locking for Azure. Use GCS with object versioning for GCP. Use Terraform Cloud/HCP Terraform when you need a managed control plane with RBAC, audit logs, and policy enforcement as a service.
• State encryption: Enable server-side encryption on the S3 bucket (AES-256 or KMS). Enable bucket versioning so state can be rolled back after accidental deletion. Enable S3 Object Lock with Compliance mode and a 30-day retention on production state buckets.
• State isolation per environment: Never share a state file between dev and prod. Each environment gets its own backend configuration with its own S3 prefix: s3://company-tfstate/prod/networking/terraform.tfstate.
• Workspaces caveat: Terraform workspaces share a single backend configuration and are appropriate for ephemeral feature environments (e.g., per-PR preview environments). They are NOT appropriate for dev/staging/prod isolation because they share provider configurations and make it easy to accidentally apply to the wrong environment. Use separate root modules for environment isolation.
• State manipulation guardrails: terraform state mv and terraform state rm must require two-person approval in production. Log all state operations. Never run terraform force-unlock without confirming the lock-holding process is dead.

4. Implement Variable Hierarchy and Secret Management

Variable management done wrong creates either brittle or insecure infrastructure:

• Variable tiering: Define variables at three levels: (a) module defaults for safe, overridable values; (b) environment-specific terraform.tfvars checked into version control for non-sensitive values; (c) sensitive values injected at runtime from a secrets manager, never committed to Git.
• Sensitive variable handling: Mark variables containing credentials, API keys, or tokens with sensitive = true in Terraform. This redacts them from terraform plan and terraform apply output. Even with this flag, sensitive values appear in state -- which is why state encryption is mandatory.
• Secret injection patterns:
  • CI/CD-injected: GitHub Actions OIDC with workload identity federation -- no static credentials stored anywhere. The pipeline assumes an IAM role with sts:AssumeRoleWithWebIdentity scoped to the specific repository and branch.
  • Runtime secret reference: Use data "aws_secretsmanager_secret_version" to pull secrets at plan/apply time. The secret ARN is a non-sensitive variable; the secret value is never in tfvars.
  • Vault integration: Use the Vault provider with AppRole or Kubernetes auth to pull secrets dynamically during apply.
• Variable validation blocks: Use Terraform validation blocks to catch invalid inputs at plan time, before any API calls. For example, validate that environment is one of ["dev", "staging", "prod"], that instance types match an approved list, or that CIDR blocks don't overlap with reserved ranges.

5. Write Production-Ready Resource Configurations

Every resource configuration must be immediately deployable, not a placeholder:

• Resource tagging strategy: Enforce mandatory tags via a locals block that merges module-level defaults with resource-specific overrides. Required tags: Environment, Team, Service, ManagedBy = "terraform", TerraformModule, CostCenter. Use AWS Tag Policies or Azure Policy to deny resources missing mandatory tags.
• Lifecycle blocks: Use prevent_destroy = true on stateful resources (RDS instances, S3 buckets with data, EKS clusters). Use create_before_destroy = true for resources that must have zero-downtime replacement (Auto Scaling Groups, Load Balancer target groups). Use ignore_changes sparingly and document every usage -- ignore_changes = [ami] is appropriate for ASGs where the AMI is managed by an image pipeline, not Terraform.
• Data source vs. hardcoded values: Never hardcode AMI IDs, VPC IDs, subnet IDs, or account IDs. Use data sources (data "aws_ami", data "aws_vpc") or remote state outputs (data "terraform_remote_state") to look up existing resources dynamically. Hardcoded IDs break when resources are recreated.
• For_each over count: Use for_each with a map or set of strings when creating multiple similar resources. count creates indexed resources (aws_subnet.this[0]) -- removing an element from the middle of the list triggers destructive replacements of all subsequent resources. for_each creates named resources (aws_subnet.this["us-east-1a"]) -- removals are surgical.
• Output discipline: Every module must output the IDs, ARNs, and names of every resource it creates. Consumers of the module should never need to reconstruct ARNs from interpolation -- the module provides them. Mark sensitive outputs with sensitive = true.

6. Build the IaC CI/CD Pipeline

A plan that nobody reviews is a plan that will eventually cause an outage:

• Pipeline stages:

  1. terraform fmt -check -- fails if code is not canonical format (enforces style in PRs)
  2. terraform validate -- syntax and provider schema validation without API calls
  3. Static analysis with Checkov or tfsec -- catches security misconfigurations (open security groups, unencrypted buckets, missing logging) before they reach AWS
  4. terraform plan against the target environment -- saves plan output as a binary artifact for the apply stage
  5. OPA/Sentinel policy evaluation against the saved plan -- enforces organizational policies (e.g., no public S3 buckets, all RDS instances must have deletion protection, only approved instance families)
  6. Manual approval gate for prod (automated apply for dev/staging)
  7. terraform apply consuming the saved plan file -- ensures what was reviewed is exactly what gets applied
  8. Post-apply smoke tests -- verify expected outputs and resource health

• Plan/apply separation: Always pass the plan file to apply: terraform apply tfplan.binary. This guarantees no drift between the reviewed plan and the applied changes. Never run terraform apply without a saved plan in CI.

• Atlantis vs. Terraform Cloud: Atlantis is self-hosted, open-source, and integrates with GitHub/GitLab PR comments (atlantis plan, atlantis apply). It works well for teams that want control over the runner. Terraform Cloud provides managed runners, a private registry, SSO, and audit logs at the cost of vendor lock-in and pricing per resource.

• Drift detection: Schedule a terraform plan on a cron (every 4-6 hours) against all production state files. Alert on any non-empty plan output. Drift indicates either a manual change that must be codified or a resource modified by an external system. Never let drift accumulate silently.

7. Implement Policy-as-Code and Compliance Controls

IaC without policy enforcement is an honor system:

• Checkov rules to enforce by default: CKV_AWS_18 (S3 access logging), CKV_AWS_20 (S3 public ACL blocked), CKV_AWS_23 (RDS deletion protection), CKV_AWS_53 (ELB deletion protection), CKV_AWS_119 (DynamoDB point-in-time recovery), CKV_AWS_145 (S3 KMS encryption). Add these to your CI pipeline as hard failures.
• OPA policy example structure: Write Rego policies that evaluate the terraform plan JSON output (terraform show -json tfplan.binary). Policies should check resource_changes for after values. A policy blocking public S3 buckets evaluates resource_changes[_].change.after.acl == "public-read" and returns a deny with a descriptive message.
• Required provider version pinning: Every root module must pin provider versions with pessimistic constraints: version = "~> 5.0" allows minor version upgrades (5.x) but not major. Required Terraform version must also be pinned: required_version = ">= 1.6.0, < 2.0.0". Use .terraform.lock.hcl committed to version control to pin exact provider checksums.
• Module pinning in production: Production root modules must reference modules via exact Git tags or registry versions. source = "terraform-aws-modules/vpc/aws" with version = "5.1.2" is correct. version = "~> 5.0" is acceptable in dev but risky in prod because a minor-version module update could change resource behavior.

8. Document, Test, and Establish the Operational Model

IaC without tests creates false confidence:

• Testing pyramid for IaC:
  • Unit: terraform validate + static analysis (runs in seconds, catches syntax and obvious misconfigs)
  • Integration: Terratest or pytest-terratest -- actually creates real cloud resources in a dedicated test account, runs assertions, then destroys resources. Use for module validation before publishing new versions.
  • End-to-end: Apply full environment stack in a staging account and run application-level smoke tests
• Terratest pattern: Write Go tests that call terraform.InitAndApply(), assert outputs match expected values, make HTTP calls to verify endpoints are reachable, then call terraform.Destroy() in a deferred cleanup. Gate module releases on passing Terratest results.
• Architecture Decision Records: Store ADRs in infra/docs/adr/ as numbered Markdown files (001-state-backend-choice.md, 002-module-versioning-strategy.md). Each ADR documents: status (proposed/accepted/deprecated), context, decision, consequences. ADRs prevent the same debates from recurring and explain why the current structure looks the way it does.
• Runbooks: Every module must have a README.md with: purpose, inputs table, outputs table, usage example, known limitations, and upgrade notes between major versions. Generated documentation via terraform-docs reduces the maintenance burden -- run it as a pre-commit hook or CI check.

Output Format

When responding to a user's IaC question, produce output in this structure:

Situation Assessment

Recommended Structure

(Provide the directory tree and explain each directory's role)

Configuration Artifacts

Provide complete, deployable files with inline comments explaining every non-obvious setting. Never use placeholder values.

Backend configuration (backend.tf):

terraform {
  backend "s3" {
    # Real values -- not placeholders
    bucket         = "company-tfstate-prod-us-east-1"
    key            = "prod/networking/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789:key/mrk-abc123"
  }
}

Provider and version pins (versions.tf):

terraform {
  required_version = ">= 1.6.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.31"
    }
  }
}

Decision Matrix

Security Controls Checklist

• State bucket encrypted with KMS CMK
• State bucket versioning enabled with 30-day Object Lock
• DynamoDB state lock table with encryption
• OIDC workload identity for CI -- no static credentials
• sensitive = true on all password/key variables
• Checkov integrated in CI with hard-fail on critical rules
• prevent_destroy = true on all stateful resources in prod
• Mandatory tags enforced via locals + provider default_tags

Pipeline Configuration

Provide the complete CI workflow file (GitHub Actions, GitLab CI, or CircleCI) with all stages.

Rules

1. Never share state files between environments. Dev, staging, and prod must each have their own backend key path and ideally their own S3 bucket. Sharing state is the #1 cause of accidental production changes during development activity.

2. Never use count for resources that might be removed from the middle of a list. Use for_each with a map keyed by a stable, meaningful identifier (availability zone name, service name). Index-based addressing (resource[0], resource[1]) causes destructive replacement cascades when the list order changes.

3. Never commit secrets to tfvars or anywhere in the repository. All passwords, API keys, and tokens must come from environment-injected secrets (OIDC, Vault, Secrets Manager data sources). Failing this rule once can expose credentials in Git history permanently, requiring rotation of every affected secret.

4. Always pin provider versions with pessimistic constraints and commit the lock file. Unpinned providers cause silent, unreproducible plan differences between developers and CI. The .terraform.lock.hcl file must be committed and its changes reviewed like any other code change.

5. Always run terraform plan before terraform apply and save the plan as an artifact. The saved plan file passed to apply guarantees that exactly the reviewed changes are applied. Ad-hoc terraform apply without a plan file is prohibited in staging and production environments.

6. Never use terraform workspace for environment isolation. Workspaces share backend configuration and provider credentials, making cross-environment accidental applies too easy. Use separate root modules in separate directories for each environment.

7. Always use for_each with module calls when creating multiple similar infrastructure stacks (e.g., multiple microservice ECS services). This enables adding and removing individual stacks without affecting others and produces named resources in state that are easy to identify.

8. Never ignore Checkov or tfsec findings without a documented suppression comment. Each suppression must include the rule ID, the reason it does not apply, and the author. Example: #checkov:skip=CKV_AWS_20:Internal bucket -- not reachable from internet, ACL restriction not applicable. Undocumented suppressions are treated as findings in audit.

9. Always implement drift detection as a scheduled pipeline job. Manual changes applied outside Terraform (ClickOps, AWS CLI, auto-scaling events that modify resource attributes) accumulate silently. Drift must be detected and either codified or reverted within one business day.

10. Never write a module that cannot be used in isolation with a minimal main.tf example. Every module must include a examples/basic/ directory with a working, deployable example. This enforces that modules have clean interfaces and prevents implicit coupling to the calling root module's state.

Edge Cases

Importing Existing Infrastructure

When an organization has years of manually provisioned AWS resources that must come under Terraform management without destroying and recreating them:

• Use Terraform 1.5+ import blocks for a declarative, reviewable import process. Write the resource configuration, add an import block with the resource address and cloud ID, run terraform plan to verify the import produces no changes, then terraform apply to write the resource into state.
• For large-scale imports (hundreds of resources), use Terraformer (terraformer import aws --resources=vpc,subnet,sg --regions=us-east-1) to generate a baseline. Treat generated code as a draft -- it will contain hardcoded values and redundant attributes that need cleanup before committing.
• After import, run terraform plan and expect a non-empty plan for attributes that Terraform manages differently from the cloud default (tags, lifecycle policies, etc.). Work through each diff systematically -- update the configuration to match existing state where appropriate, or let Terraform bring the resource into the desired state.
• Import one resource type at a time. Do not attempt to import an entire VPC and all its children in one apply. Work from the outside in: VPC first, then subnets, then route tables, then security groups.

Multi-Region Infrastructure

Deploying identical infrastructure in multiple AWS regions from a single Terraform configuration:

• Use provider aliases to configure multiple regional providers in a single root module: provider "aws" { alias = "us_west_2"; region = "us-west-2" }. Pass the provider to module calls with providers = { aws = aws.us_west_2 }.
• For completely independent regional deployments (active-active), prefer separate root modules per region with separate state files. This limits blast radius and allows per-region apply cadences. Accept the duplication.
• Never use for_each on provider configurations. Provider meta-argument for_each is not supported. If you need 6 regional deployments, you need either 6 explicit provider aliases or 6 separate root modules.
• Global resources (IAM roles, Route53 hosted zones, CloudFront distributions) must be in a dedicated global/ root module with a single state file. Reference global resource ARNs via remote state outputs in regional modules.

Monolithic Module Refactoring

When a large team has accumulated a 10,000-line main.tf monolith that needs to be split into modules without destroying live infrastructure:

• Refactoring Terraform modules is a state migration operation, not just a code refactoring. Moving a resource from one module path to another changes its state address -- Terraform will destroy the old address and create a new one unless you perform a terraform state mv.
• Plan the target module structure before touching any code. Map every current resource address to its target address. Document the full list of terraform state mv commands before executing any of them.
• Run terraform state mv commands against a local copy of the state (pulled with terraform state pull > local.tfstate) and dry-run the resulting state with terraform plan -state=local.tfstate before pushing back.
• Refactor in stages: extract one module at a time, validate with plan showing zero changes, apply the state migration, then move to the next module. Never attempt to migrate all resources in one operation.
• Use moved blocks (Terraform 1.1+) instead of CLI state mv where possible. moved blocks are code-reviewable, version-controlled, and automatically applied during terraform apply without a separate CLI operation.

Ephemeral Preview Environments

Creating per-PR temporary environments that are automatically destroyed when the PR closes:

• Use Terraform workspaces for this use case (the one legitimate workspace use case). Each PR creates a workspace named after the PR number: terraform workspace new pr-1234. CI applies the full stack to this workspace and posts the output URLs as PR comments.
• Set aggressive auto_destroy schedules. Workspaces older than 48 hours without recent commit activity should be automatically destroyed by a cleanup job. Orphaned preview environments are a significant cost leak.
• Limit preview environment scope to stateless compute resources (ECS tasks, Lambda functions, API Gateway stages). Never create RDS instances, Elasticsearch clusters, or other slow/expensive stateful resources in preview environments -- use shared dev databases with namespace isolation instead.
• Use terraform output -json to extract dynamic URLs (ALB DNS names, CloudFront domains) and post them to the PR via GitHub API. This makes preview environments discoverable without checking CI logs.
• Implement resource naming that includes the workspace name to prevent collision: "${var.service_name}-${terraform.workspace}-api". Without workspace in the name, resources from different PRs will collide in the same AWS account.

Secrets Rotation Without Downtime

When a database password or API key managed by Terraform needs rotation without application downtime:

• Terraform-managed secrets can create a chicken-and-egg problem: updating the secret in Secrets Manager updates the Terraform state, but if the application reads the secret at startup only, it will not pick up the rotation without a restart.
• For RDS passwords, use AWS Secrets Manager rotation with a Lambda rotator. The Lambda performs a two-phase rotation (set pending, update DB user, set current) that keeps one valid version active throughout. Terraform manages the rotation schedule configuration (rotation_rules) but the rotation itself is handled out-of-band.
• Never update a secret resource in Terraform by changing the secret_string value directly if the application is using the ARN reference. The update will succeed in state but the application connection pool will fail until restart. Coordinate the Terraform apply with a rolling application restart in the same deployment pipeline.
• For TLS certificates, use create_before_destroy = true on aws_acm_certificate and let the ACM validation complete before the load balancer listener is updated. The lifecycle block ensures the new certificate is valid before the old one is removed from the listener.

Terragrunt for DRY Multi-Environment Configuration

When pure Terraform root module duplication across environments becomes unmanageable (identical configurations with only account ID, region, and tier-specific sizing as differences):

• Terragrunt wraps Terraform and adds a terragrunt.hcl configuration layer that generates backend configurations dynamically, passes common inputs from parent terragrunt.hcl files, and orchestrates multi-module deploys with dependency ordering.
• Use Terragrunt when: you have 5+ environments, backend configuration is identical except for path/account/region, and you find yourself copy-pasting the same provider and backend blocks across directories.
• Do NOT use Terragrunt when: your team is not already proficient with Terraform, you only have 2-3 environments, or your organization uses Terraform Cloud (which already handles these concerns). Terragrunt adds a layer of abstraction that requires understanding both tools.
• The canonical Terragrunt directory structure uses a three-level hierarchy: root.hcl (org-wide defaults), account.hcl (account/subscription-level config), region.hcl (region-level config), and per-module terragrunt.hcl files that inherit all parent config. The read_terragrunt_config() function merges these layers.

Compliance and Audit Requirements (SOC 2, HIPAA, PCI-DSS)

When IaC operates in regulated environments with audit obligations:

• Every terraform apply must be traceable to: who triggered it, which version of code was applied, what the plan showed, and what the outcome was. Terraform Cloud and Atlantis both provide audit logs. For self-managed pipelines, archive the plan output and apply log to an immutable S3 bucket with Object Lock.
• PCI-DSS requires that changes to production infrastructure go through a formal change management process with pre-approval. Implement a required manual approval step in the CD pipeline that triggers a notification to a compliance Slack channel and requires approval from a named approver.
• SOC 2 Type II requires evidence of access controls on infrastructure management. Use OIDC for CI pipelines (eliminates standing credentials), separate IAM roles per environment (apply role has write access to prod; plan role is read-only), and enforce MFA on all human IAM users who can assume apply roles.
• HIPAA-covered IaC must ensure that PHI is never written to Terraform state. This means never using data sources that return PHI, never storing PHI in resource tags, and encrypting state at rest with a CMK under the covered entity's control. Document encryption key management in your risk assessment.

Example

Input: "We have a team of 8 engineers and are moving to AWS. We have an existing ECS-based application running manually provisioned infrastructure (VPCs, ECS clusters, RDS, ALBs all ClickOps'd). We need to bring it under Terraform management with dev/staging/prod environments. We use GitHub Actions for CI."

Output:

Situation Assessment

Recommended Directory Structure

infra/
├── modules/
│   ├── networking/
│   │   ├── main.tf          # VPC, subnets (public/private/data), IGW, NAT GW
│   │   ├── variables.tf
│   │   ├── outputs.tf       # vpc_id, subnet_ids, nat_gateway_ids
│   │   └── README.md
│   ├── ecs-service/
│   │   ├── main.tf          # ECS task definition, service, CloudWatch logs, IAM
│   │   ├── variables.tf
│   │   ├── outputs.tf       # service_name, task_definition_arn, cloudwatch_group
│   │   └── README.md
│   ├── rds-postgres/
│   │   ├── main.tf          # DB instance, subnet group, parameter group, secret
│   │   ├── variables.tf
│   │   └── outputs.tf       # db_endpoint, secret_arn
│   └── alb/
│       ├── main.tf          # ALB, target groups, listeners, ACM cert
│       ├── variables.tf
│       └── outputs.tf       # alb_dns_name, target_group_arns
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── versions.tf
│   │   ├── backend.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   │   └── (same structure)
│   └── prod/
│       └── (same structure)
├── .github/
│   └── workflows/
│       └── terraform.yml
├── policies/
│   └── checkov-baseline.yaml   # Checkov suppress list with documented reasons
└── docs/
    └── adr/
        ├── 001-state-backend.md
        ├── 002-module-versioning.md
        └── 003-import-strategy.md

Phase 1: State Backend Bootstrap

Create this once manually (bootstrapping is the one acceptable ClickOps action):

# bootstrap/main.tf -- apply once, then never touch again
# This creates the S3 bucket and DynamoDB table that all other state uses

resource "aws_s3_bucket" "tfstate" {
  bucket = "company-tfstate-us-east-1"

  # Prevent accidental deletion of state bucket -- this is catastrophic
  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.tfstate.arn
    }
    # Enforce KMS -- deny requests using S3-managed keys
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "tfstate" {
  bucket                  = aws_s3_bucket.tfstate.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_kms_key" "tfstate" {
  description             = "KMS key for Terraform state encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true  # Auto-rotate annually

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Allow account root"
        Effect = "Allow"
        Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" }
        Action = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow GitHub Actions OIDC role"
        Effect = "Allow"
        Principal = { AWS = aws_iam_role.github_actions.arn }
        Action = ["kms:GenerateDataKey", "kms:Decrypt"]
        Resource = "*"
      }
    ]
  })
}

resource "aws_dynamodb_table" "tfstate_lock" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"  # No capacity planning required
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  point_in_time_recovery {
    enabled = true
  }

  server_side_encryption {
    enabled = true
  }
}

Production Root Module (environments/prod/main.tf)

# environments/prod/main.tf
# Production environment -- all changes require PR approval and CI apply gate
# Last reviewed: see git log

locals {
  environment = "prod"
  region      = "us-east-1"

  # Mandatory tags applied to all resources via provider default_tags
  # Add resource-specific tags on individual resources, not here
  common_tags = {
    Environment = local.environment
    ManagedBy   = "terraform"
    TerraformRepo = "github.com/company/infra"
    CostCenter  = "platform-engineering"
    Team        = "platform"
  }
}

module "networking" {
  source = "../../modules/networking"

  environment          = local.environment
  vpc_cidr             = "10.0.0.0/16"

  # Three AZs for production HA -- dev uses 2 to save NAT Gateway cost
  availability_zones   = ["us-east-1a", "us-east-1b", "us-east-1c"]

  # /20 = 4096 IPs per subnet -- sized for 3-year growth
  private_subnet_cidrs = ["10.0.16.0/20", "10.0.32.0/20", "10.0.48.0/20"]
  public_subnet_cidrs  = ["10.0.0.0/24",  "10.0.1.0/24",  "10.0.2.0/24"]
  data_subnet_cidrs    = ["10.0.64.0/24", "10.0.65.0/24", "10.0.66.0/24"]

  # One NAT gateway per AZ -- costs ~$32/month each but required for HA
  # Dev uses single NAT to save ~$64/month
  single_nat_gateway   = false

  tags = local.common_tags
}

module "alb" {
  source = "../../modules/alb"

  environment      = local.environment
  vpc_id           = module.networking.vpc_id
  public_subnet_ids = module.networking.public_subnet_ids
  certificate_arn  = data.aws_acm_certificate.api.arn

  # ALB access logs for compliance and debugging
  access_logs_bucket = aws_s3_bucket.alb_logs.bucket

  deletion_protection = true  # prevent_destroy equivalent for ALB

  tags = local.common_tags
}

module "rds_postgres" {
  source = "../../modules/rds-postgres"

  environment          = local.environment
  identifier           = "prod-payments-pg"
  engine_version       = "15.4"
  instance_class       = "db.r7g.large"  # 2 vCPU, 16 GB -- right-sized for current load
  allocated_storage    = 100             # GB -- monitors via CloudWatch StorageSpace alarm
  max_allocated_storage = 500            # GB -- autoscaling upper bound

  db_subnet_group_ids  = module.networking.data_subnet_ids
  vpc_id               = module.networking.vpc_id

  # Allow inbound from ECS tasks only -- not from public subnets
  allowed_security_group_ids = [module.ecs_service.task_security_group_id]

  # These are fetched from Secrets Manager -- not in tfvars
  # The secret is created by this module; the value is set manually on first deploy
  # Rotation is managed by an AWS Lambda rotator
  manage_master_password = true  # AWS-managed password rotation

  # Production requires multi-AZ for automatic failover
  multi_az              = true
  deletion_protection   = true
  skip_final_snapshot   = false
  final_snapshot_identifier = "prod-payments-pg-final-${formatdate("YYYY-MM-DD", timestamp())}"

  backup_retention_period = 35  # Days -- maximum for RDS automated backups
  backup_window           = "03:00-04:00"  # UTC -- low-traffic window
  maintenance_window      = "sun:04:00-sun:05:00"

  # PostgreSQL parameter group tuning for OLTP workload
  parameters = {
    "max_connections"          = "200"
    "shared_buffers"           = "{DBInstanceClassMemory/32768}MB"
    "effective_cache_size"     = "{DBInstanceClassMemory/16384}MB"
    "log_min_duration_statement" = "1000"  # Log queries > 1 second
    "log_connections"          = "1"
    "log_disconnections"       = "1"
  }

  tags = local.common_tags
}

module "ecs_service" {
  source = "../../modules/ecs-service"

  environment          = local.environment
  service_name         = "payments-api"
  cluster_arn          = aws_ecs_cluster.prod.arn
  container_image      = var.container_image  # Injected by application CD pipeline
  container_port       = 8080
  cpu                  = 1024  # 1 vCPU
  memory               = 2048  # 2 GB

  # Fetch DB secret ARN from RDS module -- not hardcoded
  secrets = {
    DATABASE_URL = module.rds_postgres.connection_secret_arn
  }

  private_subnet_ids   = module.networking.private_subnet_ids
  vpc_id               = module.networking.vpc_id
  target_group_arn     = module.alb.api_target_group_arn

  desired_count        = 3   # Minimum for AZ redundancy
  max_capacity         = 20  # Auto Scaling upper bound

  # Health check -- must pass before ALB routes traffic
  health_check_path    = "/health"
  health_check_grace_period_seconds = 60

  tags = local.common_tags
}

# Look up existing ACM cert -- managed separately (auto-renewed by ACM)
data "aws_acm_certificate" "api" {
  domain      = "api.company.com"
  statuses    = ["ISSUED"]
  most_recent = true
}

Backend Configuration (environments/prod/backend.tf)

terraform {
  backend "s3" {
    bucket         = "company-tfstate-us-east-1"
    key            = "prod/main/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789012:key/mrk-0123456789abcdef0"
    # Role assumed during CI apply -- read-only plan role is different
    role_arn       = "arn:aws:iam::123456789012:role/terraform-prod-apply"
  }
}

…(truncated)