Terraform GCP Scaffolder

Terraform GCP scaffolder

Read references/conventions.md for naming and labels.

Layout

infra/
  modules/
    bq-dataset/
    cloud-run-job/
    service-account/
  envs/
    dev/    main.tf  terraform.tfvars  backend.tf
    prod/   main.tf  terraform.tfvars  backend.tf

One state file per environment, remote backend in GCS with versioning on. Never share state between dev and prod, and never let a module create its own project.

# envs/prod/backend.tf
terraform {
  required_version = ">= 1.9"
  backend "gcs" {
    bucket = "acme-tfstate-prod"
    prefix = "data-platform"
  }
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0"
    }
  }
}

BigQuery dataset module

# modules/bq-dataset/main.tf
resource "google_bigquery_dataset" "this" {
  dataset_id                 = var.dataset_id
  project                    = var.project_id
  location                   = var.location
  description                = var.description
  delete_contents_on_destroy = false

  default_partition_expiration_ms = var.partition_expiration_days == null ? null : var.partition_expiration_days * 86400000

  labels = merge(var.labels, {
    env    = var.env
    domain = var.domain
    owner  = var.owner
  })
}

resource "google_bigquery_dataset_iam_member" "readers" {
  for_each   = toset(var.reader_members)
  project    = var.project_id
  dataset_id = google_bigquery_dataset.this.dataset_id
  role       = "roles/bigquery.dataViewer"
  member     = each.value
}

resource "google_bigquery_dataset_iam_member" "writers" {
  for_each   = toset(var.writer_members)
  project    = var.project_id
  dataset_id = google_bigquery_dataset.this.dataset_id
  role       = "roles/bigquery.dataEditor"
  member     = each.value
}

Grant at the dataset level, not the project level. roles/bigquery.dataViewer on a project is how an intern reads payroll.

Service account with least privilege

resource "google_service_account" "dataform" {
  account_id   = "sa-dataform-${var.env}"
  display_name = "Dataform execution, ${var.env}"
  project      = var.project_id
}

# Job runner at project level, data access scoped per dataset
resource "google_project_iam_member" "dataform_job_user" {
  project = var.project_id
  role    = "roles/bigquery.jobUser"
  member  = "serviceAccount:${google_service_account.dataform.email}"
}

resource "google_bigquery_dataset_iam_member" "dataform_writes_marts" {
  project    = var.project_id
  dataset_id = "mart_retail"
  role       = "roles/bigquery.dataEditor"
  member     = "serviceAccount:${google_service_account.dataform.email}"
}

Never create service account keys in Terraform. Use workload identity federation for CI and the attached service account for Cloud Run and Composer. If someone asks for a key file, ask what they are authenticating from first, because the answer is almost always something that supports WIF.

Cloud Run job on a schedule

resource "google_cloud_run_v2_job" "loader" {
  name     = "job-${var.name}-${var.env}"
  location = var.region
  project  = var.project_id

  template {
    task_count = 1
    template {
      service_account = google_service_account.job.email
      max_retries     = 3
      timeout         = "1800s"
      containers {
        image = var.image
        resources {
          limits = { cpu = "1", memory = "2Gi" }
        }
        dynamic "env" {
          for_each = var.secrets
          content {
            name = env.key
            value_source {
              secret_key_ref {
                secret  = env.value
                version = "latest"
              }
            }
          }
        }
      }
    }
  }
}

resource "google_cloud_scheduler_job" "trigger" {
  name     = "sched-${var.name}-${var.env}"
  schedule = var.cron
  time_zone = "UTC"
  project  = var.project_id
  region   = var.region

  http_target {
    http_method = "POST"
    uri         = "https://${var.region}-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/${var.project_id}/jobs/${google_cloud_run_v2_job.loader.name}:run"
    oauth_token {
      service_account_email = google_service_account.scheduler.email
    }
  }
}

Plan is a read, apply is not

Read references/execution-model.md. Run terraform plan and read the output, specifically the destroy list, because a plan that quietly re-creates a stateful resource is the finding that matters. Never run terraform apply, and never run terraform state subcommands that mutate state.

When scaffolding IAM for an agent that will use the BigQuery MCP server, grant roles/mcp.toolUser, roles/bigquery.jobUser, and roles/bigquery.dataViewer to a dedicated service account, never to a human's identity, and pair it with a deny policy on read-write MCP tool use.

Review checklist

• Every resource carries env, owner, domain labels.
• No roles/editor, no roles/owner, no allUsers, no primitive roles at all.
• No service account keys.
• prevent_destroy on state buckets, production datasets, and Composer.
• Variables have types and descriptions. variable "x" {} with no type is a bug.
• for_each over count for anything keyed by name, so adding one entry does not re-create the others.
• Provider version pinned with ~>, not floating.