Deploy Dev

Dev Environment Architecture

Dev environments (dev1-4) run in the staging AWS account.

Shared resources (when create_* flags are false in tfvars):

• S3 bucket: staging-metr-inspect-data
• EventBridge bus: staging-inspect-ai-api
• EKS cluster: staging-eks-cluster
• ALB, VPC, subnets

Dedicated per environment:

• Warehouse DB: {env_name}-inspect-ai-warehouse
• API service: {env_name}-inspect-ai-api
• Lambda functions: {env_name}-inspect-ai-*
• ECR repos: {env_name}/inspect-ai/*
• CloudWatch logs

Config lives in terraform/terraform.tfvars (gitignored — set to your target environment before deploying).

Note: Commands below may require AWS credentials for the staging account. Set AWS_PROFILE or configure credentials as appropriate for your environment.

Targeted Deploy

When you only changed one component:

tofu -chdir=terraform apply \
  -var-file="terraform.tfvars" -target=module.<target> -auto-approve

Common targets:

Full Deploy

# Preview changes
tofu -chdir=terraform plan -var-file="terraform.tfvars"

# Apply all changes
tofu -chdir=terraform apply -var-file="terraform.tfvars"

Runner Image Rebuilds

Terraform automatically tracks uv.lock, pyproject.toml, and Python source files to determine when the runner Docker image needs rebuilding (see terraform/modules/runner/ecr.tf). tofu apply is the correct way to rebuild runner images — it produces deterministic sha256.<hash> image tags.

Do NOT use scripts/dev/build-and-push-runner-image.sh for deploying to dev environments. That script is only for local hawk local testing.

Docker Registry Authentication

Before tofu apply can build and push images, you must be logged into:

1. ECR: Install the docker-credential-ecr-login helper and configure Docker to use it
2. DHI (hardened images) registry: docker login dhi.io

Lambda Lock Files

After changing Python dependencies (pyproject.toml or uv.lock), run ./scripts/dev/uv-lock-all.sh to update lock files in ALL terraform modules. Terraform modules have their own lock files that must stay in sync with the root.

Database Migrations

See the db-migrations skill for full details.

Quick command:

DATABASE_URL=$(tofu -chdir=terraform output \
  -var-file="terraform.tfvars" -raw warehouse_database_url_admin) \
  alembic upgrade head

Verify with Smoke Tests

See the smoke-tests skill for full details.

Quick command:

unset HAWK_MODEL_ACCESS_TOKEN_ISSUER
scripts/dev/create-smoke-test-env.py env/smoke-dev2 --terraform-dir terraform
set -a && source env/smoke-dev2 && set +a && \
  pytest tests/smoke -m smoke --smoke -vv -n 5

Complete Workflow

The typical deploy-and-verify loop:

1. Make code changes
2. Run local tests: pytest tests/{package} -n auto -vv
3. Run quality checks: ruff check . && ruff format . --check && basedpyright .
4. Deploy to dev: tofu apply -target=module.X
5. Run migrations if needed (see Database Migrations section above)
6. Run smoke tests to verify
7. Commit and push

Choosing Between Shared and Dedicated Infrastructure

Dev environments default to reusing staging infra. To create dedicated resources, change these flags in your tfvars:

# Shared (default for dev environments)
create_s3_bucket       = false
create_eventbridge_bus = false
create_eks_resources   = false

# Dedicated (set to true + provide unique names)
create_s3_bucket       = true
s3_bucket_name         = "dev5-metr-inspect-data"
create_eventbridge_bus = true
eventbridge_bus_name   = "dev5-inspect-ai-api"
create_eks_resources   = true

Use shared (recommended): Cheaper, simpler, sufficient for most dev work.

Use dedicated: When testing infrastructure changes, or when you need full isolation from other devs.

Shared Mode Implications (create_eks_resources=false)

When using shared infrastructure, dev environments reuse staging's k8s namespace, ClusterRole, and ClusterRoleBinding. This means:

• Runner pods use the staging k8s service account and RBAC bindings
• Changes to k8s auth patterns (e.g., in-cluster config, service account permissions) must be tested in shared mode — they may work on staging (which has its own dedicated ClusterRoleBinding) but fail on dev environments that rely on staging's bindings
• The Helm chart creates a RoleBinding in the sandbox namespace only — ClusterRoleBinding for the runner namespace comes from terraform (only when create_eks_resources=true)

Useful Log Groups

Tail logs: aws logs tail <log-group> --since 30m --format short

Troubleshooting

Converted and distributed by TomeVault — claim your Tome and manage your conversions.