Terraform For Data Infra

Terraform for Data Infrastructure

When to use

• Provisioning data platform resources as code (warehouses, buckets, IAM, orchestration, streaming).
• Structuring reusable modules and multiple environments (dev/staging/prod).
• Setting up remote state and safe plan/apply workflows.
• Do NOT use for in-warehouse data modeling (use dbt) or app infra unrelated to data.

Workflow

- [ ] Use remote state with locking (S3+DynamoDB / GCS / Terraform Cloud)
- [ ] Factor reusable pieces into modules; parameterize per environment
- [ ] Separate environments via workspaces or per-env state, not copy-paste
- [ ] Always review terraform plan before apply
- [ ] Manage least-privilege IAM/roles as code; keep secrets out of state

1. Remote state + locking. Never keep state locally on a team; use a locked remote backend so concurrent applies can't corrupt it.
2. Modules. Encapsulate a warehouse, bucket+lifecycle, or role set as a module and instantiate it per environment with variables.
3. Environments. Use workspaces or separate state per env; keep one codebase, varying only inputs — mirrors the "same code, different target" data-CI/CD rule.
4. Plan before apply. Review the plan; data resources (warehouses, roles, buckets) are high-blast-radius.
5. Least privilege as code. Define IAM/roles/grants in Terraform; avoid embedding secrets in state.

Patterns

Environment via variables + module:

module "warehouse" {
  source        = "./modules/snowflake-warehouse"
  name          = "etl_${var.env}"
  size          = var.env == "prod" ? "LARGE" : "XSMALL"
  auto_suspend  = 60
  auto_resume   = true
}

Remote state backend (S3 + lock table):

terraform {
  backend "s3" {
    bucket         = "acme-tfstate"
    key            = "data/${terraform.workspace}/terraform.tfstate"
    dynamodb_table = "tf-locks"
  }
}

Keep secrets out of state — source them from a secrets manager (aws_secretsmanager_secret_version, Vault) at apply time; treat state as sensitive and encrypt the backend.

Common pitfalls

• Local state on a team — lost/corrupted state and drift; use a locked remote backend.
• Copy-pasting per environment — configs drift; use modules + variables.
• apply without reviewing plan — accidental warehouse resize or role deletion.
• Secrets in variables/state — plaintext credentials leak; use a secrets manager and mark values sensitive.
• Over-broad IAM — grant least privilege; wildcards become audit findings.
• Managing data content in Terraform — it manages infrastructure, not rows; keep data transformations in dbt/pipelines.