AWS Investigation

AWS Infrastructure Investigation

Investigate AWS infrastructure issues affecting Buildkite build agents. Covers EC2 instances, AutoScaling Groups, the autoscaling Lambda, and related resources.

Prerequisites

• AWS CLI installed (brew install awscli)
• AWS SSO profile mockserver-build configured (SSO region: eu-west-2)
• Active SSO session: aws sso login --profile mockserver-build
• Corporate TLS proxy (if applicable): export AWS_CA_BUNDLE=$NODE_EXTRA_CA_CERTS (only if NODE_EXTRA_CA_CERTS is set)
• macOS + Python 3.14 + Homebrew: if you get pyexpat symbol errors, export DYLD_LIBRARY_PATH=/opt/homebrew/opt/expat/lib

Infrastructure Overview

There are two build agent stacks. Investigate the current stack first; fall back to the legacy stack only if the current one has not been provisioned yet.

Current: Terraform-managed (eu-west-2)

Managed by terraform/buildkite-agents/ using the official Buildkite Elastic CI Stack module.

Resource names are generated by Terraform with a random suffix. To find them:

# Get ASG name from Terraform state
cd terraform/buildkite-agents
terraform output auto_scaling_group_name

# Or find ASGs with the Buildkite tag
aws autoscaling describe-auto-scaling-groups \
  --region eu-west-2 --profile mockserver-build \
  --query 'AutoScalingGroups[?contains(Tags[?Key==`Stack`].Value | [0], `buildkite-mockserver`)].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Min:MinSize,Max:MaxSize,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

Legacy: CloudFormation-managed (us-east-1)

Being replaced by the Terraform stack above. May still be active during migration.

AWS CLI Prefix

All commands require --region and --profile flags:

# Current stack (eu-west-2)
aws ... --region eu-west-2 --profile mockserver-build

# Legacy stack (us-east-1)
aws ... --region us-east-1 --profile mockserver-build

Investigation Workflow

Step 1: Determine Active Stack

Check which stack is currently running agents:

# Check current stack (eu-west-2) — look for ASGs tagged with buildkite-mockserver
aws autoscaling describe-auto-scaling-groups \
  --region eu-west-2 --profile mockserver-build \
  --query 'AutoScalingGroups[?contains(Tags[?Key==`Stack`].Value | [0], `buildkite-mockserver`)].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

# Check legacy stack (us-east-1)
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "buildkite-AgentAutoScaleGroup-VGG28FR0DE6Q" \
  --region us-east-1 --profile mockserver-build \
  --query 'AutoScalingGroups[0].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

Use whichever stack has instances (or non-zero desired capacity) for the remaining steps. Substitute the correct --region and ASG name accordingly.

Step 2: Quick Health Check

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --query 'AutoScalingGroups[0].{Desired:DesiredCapacity,Min:MinSize,Max:MaxSize,Instances:Instances[*].{ID:InstanceId,State:LifecycleState,Health:HealthStatus}}'

Expected healthy state:

• If queue is empty: Desired = 0 can be healthy (scale-to-zero)
• If queue has pending jobs: desired capacity should increase above 0 within 1-2 scaler intervals
• Active instances should be InService and Healthy

Problem indicators:

• Desired: 0 — no agents requested (scaler not seeing jobs, or Lambda not running)
• Desired > 0 but no instances — launch failures
• Instances in Pending for >5 min — launch issues
• Instances Unhealthy — failing health checks

Step 3: Check EC2 Instance Status

aws ec2 describe-instances \
  --filters "Name=tag:aws:autoscaling:groupName,Values=<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name,Type:InstanceType,Launch:LaunchTime,AZ:Placement.AvailabilityZone}'

For running instances, check system/instance status:

aws ec2 describe-instance-status \
  --instance-ids <instance-id-1> <instance-id-2> \
  --region <REGION> --profile mockserver-build

Step 4: Check Scaling Activities

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --max-items 10

Look for:

• "user request explicitly set group desired capacity" — the Lambda scaler adjusted capacity
• "an instance was taken out of service" — scale-in event
• Failed status codes — launch failures (AMI issues, capacity, subnet exhaustion)

Step 5: Check the Autoscaling Lambda

Find the scaler Lambda by listing functions with a Buildkite-related name:

aws lambda list-functions \
  --region <REGION> --profile mockserver-build \
  --query 'Functions[?contains(FunctionName, `buildkite`) && (contains(FunctionName, `scaler`) || contains(FunctionName, `caling`))].{Name:FunctionName,Runtime:Runtime,State:State,LastModified:LastModified}'

Then check its logs:

# Recent invocations (last hour)
aws logs filter-log-events \
  --log-group-name "/aws/lambda/<LAMBDA_FUNCTION_NAME>" \
  --region <REGION> --profile mockserver-build \
  --start-time $(python3 -c "import time; print(int((time.time() - 3600) * 1000))") \
  --limit 20

# Error logs (last hour)
aws logs filter-log-events \
  --log-group-name "/aws/lambda/<LAMBDA_FUNCTION_NAME>" \
  --region <REGION> --profile mockserver-build \
  --start-time $(python3 -c "import time; print(int((time.time() - 3600) * 1000))") \
  --filter-pattern "ERROR" \
  --limit 10

Step 6: Check EC2 Console Output

For instances that are running but not registering as Buildkite agents:

aws ec2 get-console-output \
  --instance-id <instance-id> \
  --region <REGION> --profile mockserver-build \
  --query 'Output' --output text

Step 7: Check Suspended ASG Processes

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build \
  --query 'AutoScalingGroups[0].SuspendedProcesses'

Note: AZRebalance is intentionally suspended to prevent killing running builds. Other suspended processes may indicate problems.

Step 8 (optional): Compare Against the Last Healthy Window

A snapshot of the current state is hard to read without a baseline. When the symptom is "something changed / got worse", compare the current state against the last healthy window — e.g. a prior good scaling cycle or a metric baseline — so a deviation is visible rather than guessed:

# Scaling activities further back, to find the last cycle that launched and
# served instances cleanly (the healthy baseline to compare the current one against)
aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name "<ASG_NAME>" \
  --region <REGION> --profile mockserver-build --max-items 40

# CPU baseline vs now for the ASG (last good window vs the suspect window)
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=AutoScalingGroupName,Value=<ASG_NAME> \
  --start-time $(python3 -c "import time; print(__import__('datetime').datetime.fromtimestamp(time.time()-86400, tz=__import__('datetime').timezone.utc).strftime('%Y-%m-%dT%H:%M:%S'))") \
  --end-time $(python3 -c "print(__import__('datetime').datetime.now(__import__('datetime').timezone.utc).strftime('%Y-%m-%dT%H:%M:%S'))") \
  --period 3600 --statistics Average Maximum \
  --region <REGION> --profile mockserver-build

Report the deviation from baseline (e.g. the last good cycle reached healthy capacity quickly whereas this cycle has stayed Pending far longer) rather than the raw current numbers alone. If no baseline is available, say so.

Failure Patterns

Emergency: Manually Scale Up Agents

If the Lambda is broken and you need agents immediately:

aws autoscaling set-desired-capacity \
  --auto-scaling-group-name "<ASG_NAME>" \
  --desired-capacity <TEMP_CAPACITY_LEQ_MAX_SIZE> \
  --region <REGION> --profile mockserver-build

Choose a temporary capacity that does not exceed the ASG MaxSize from Step 2.

Warning: The Lambda scaler may override this on its next invocation if it sees no pending jobs.

Enumerate Competing Hypotheses

Before concluding a root cause, enumerate the competing hypotheses and the evidence that rules each out (correlation is not causation — a scaling event coinciding with the symptom is not proof it caused it; e.g. "Lambda not invoking" vs "Lambda invoking but Buildkite API auth failing" vs "launch template / capacity error" are distinct causes with distinct evidence). Record the survivors and the ruled-out alternatives in root_cause.alternative_hypotheses.

Output — Structured Data Return

Return this structure in your final message:

{
  "schema": "aws-investigation/v1",
  "timestamp": "<ISO8601>",
  "active_stack": "terraform-eu-west-2 | legacy-us-east-1",
  "asg": {
    "name": "<ASG name>",
    "region": "<region>",
    "desired_capacity": 0,
    "min_size": 0,
    "max_size": "<max_size>",
    "instances": [
      {
        "instance_id": "<id>",
        "state": "InService|Pending|Terminating",
        "health": "Healthy|Unhealthy",
        "availability_zone": "<az>"
      }
    ],
    "suspended_processes": ["<process names>"]
  },
  "lambda": {
    "function_name": "<name>",
    "state": "Active|Inactive",
    "runtime": "<runtime>",
    "recent_errors": ["<error messages>"],
    "last_invocation": "<ISO8601 or null>"
  },
  "root_cause": {
    "summary": "<one-line description>",
    "detail": "<technical explanation>",
    "category": "<category from failure patterns>",
    "evidence": "<relevant log lines or CLI output>",
    "alternative_hypotheses": [
      { "hypothesis": "<competing explanation>", "ruled_out_by": "<evidence that excludes it>" }
    ]
  },
  "baseline_comparison": "<deviation from the last healthy window, or null if no baseline available>",
  "commands_run": ["<exact aws / terraform command used to gather evidence>"],
  "recommended_fix": "<actionable steps>",
  "warnings": ["<deprecation notices, capacity concerns, etc.>"]
}

After returning the JSON, provide a brief summary (2-3 lines).

Notes

• Always run Step 1 first to determine which stack is active
• The Lambda scaler logs are the most valuable data source for understanding scaling decisions
• Always check if the Buildkite agent token is still valid if agents start but don't register