Smoke Tests

Prerequisites

1. Login: hawk login (stores OAuth2 tokens in keyring — shared across all environments, only need to do this once)
2. AWS credentials: Ensure credentials are configured for the target environment (e.g., set AWS_PROFILE if needed)
3. Unset overrides: unset HAWK_MODEL_ACCESS_TOKEN_ISSUER (if previously set)

After any tofu apply: You MUST regenerate the smoke env file. Terraform outputs change (image tags, URLs, etc.) and stale env files will cause confusing failures.

Running Smoke Tests

1. Generate Environment File

For dev environments (dev1-4, using this repo's terraform):

unset HAWK_MODEL_ACCESS_TOKEN_ISSUER
scripts/dev/create-smoke-test-env.py env/smoke-dev2 --terraform-dir terraform

For staging (using the separate deploy repo):

unset HAWK_MODEL_ACCESS_TOKEN_ISSUER
scripts/dev/create-smoke-test-env.py env/smoke-staging --terraform-dir ../mp4-deploy/terraform_inspect

Note: create-smoke-test-env.py reads Terraform outputs, which may require AWS credentials.

2. Run Tests

# Using uv (outside virtual environment):
set -a && source env/smoke-<env> && set +a && \
  uv run pytest tests/smoke -m smoke --smoke -vv -n 5

# Already in virtual environment:
set -a && source env/smoke-<env> && set +a && \
  pytest tests/smoke -m smoke --smoke -vv -n 5

Or using --env-file (avoids polluting your shell):

uv run --env-file env/smoke-<env> pytest tests/smoke -m smoke --smoke -vv -n 5

• Use -n 5 for dev environments (middleman can't handle more concurrent requests)
• Use -n 10 for staging

Important: The hawk CLI reads HAWK_API_URL from the process environment — it does NOT auto-load .env files. You must either source the env file or use --env-file.

3. Run a Single Test

# Using uv:
set -a && source env/smoke-<env> && set +a && \
  uv run pytest tests/smoke/test_outcomes.py::test_model_roles --smoke -vv

# Already in virtual environment:
set -a && source env/smoke-<env> && set +a && \
  pytest tests/smoke/test_outcomes.py::test_model_roles --smoke -vv

Auth Model

Smoke tests use two separate auth mechanisms:

• API calls (eval set submission, log viewer, sample edits): OAuth2 bearer tokens from keyring (hawk login)
• Warehouse database: IAM authentication via AWS credentials — this is why AWS access must be configured even though you already logged in

Troubleshooting

Before You Debug: Verify the Basics

When smoke tests fail, check these FIRST before investigating further:

1. Env file regenerated? If you ran tofu apply since the last env file generation, regenerate it now
2. Correct environment sourced? Run echo $HAWK_API_URL — does it point to the right environment?
3. Network connectivity? Ensure Tailscale is connected. If you're running in a different AWS VPC than the target environment (e.g., running in production VPC but targeting staging), you'll need to resolve network routing — ask for help
4. Logged in? hawk auth access-token > /dev/null && echo OK — if this fails, run hawk login

Common Errors

Debugging Approach

ALWAYS find root cause before attempting fixes. When investigating failures:

1. Read error messages literally — don't skip past errors to guess at causes. If Helm says "configmaps is forbidden", the issue IS permissions, not something else
2. Gather evidence before forming hypotheses — check logs, pod status, env vars before proposing a fix
3. Verify one thing at a time — don't change multiple things and re-test

When Samples Don't Appear in Warehouse

This means the eval completed but the import pipeline failed. The investigation path is:

S3 → EventBridge → Lambda → Batch → DB

See docs/solutions/smoke-test-failures/investigating-batch-import-failures.md for the full debugging guide with commands for each stage.

Related Skills and Docs

• deploy-dev skill: Deploy code changes to dev environments before running smoke tests
• db-migrations skill: Run migrations against dev warehouse DBs (often needed after deploying schema changes)
• docs/solutions/smoke-test-failures/: Detailed investigation writeups for specific failure patterns