Ecs Diagnostics

ECS Diagnostics

When to use

Any ECS investigation where the ECS console alone is insufficient — stopped task reasons, container exit codes, OOM kills, image pull errors, networking issues, IAM permission failures, capacity provider scaling, deployment circuit breakers, service mesh issues, or exec failures.

Investigation workflow

Step 1 — Collect and triage

# Get service details and events
aws ecs describe-services --cluster <cluster> --services <service>

# Get recent task failures
aws ecs list-tasks --cluster <cluster> --service-name <service> --desired-status STOPPED
aws ecs describe-tasks --cluster <cluster> --tasks <task-arns>

# Get container logs
aws logs filter-log-events --log-group-name <log-group> --filter-pattern "ERROR"

# Get service events (last 100)
aws ecs describe-services --cluster <cluster> --services <service>  # → events[]

Step 2 — Domain deep dive

# Task definition review
aws ecs describe-task-definition --task-definition <family:revision>

# Container instance state (EC2 launch type)
aws ecs list-container-instances --cluster <cluster>
aws ecs describe-container-instances --cluster <cluster> --container-instances <arns>

# Capacity providers
aws ecs describe-capacity-providers --capacity-providers <names>

# Load balancer health
aws elbv2 describe-target-health --target-group-arn <arn>

# ECR image check
aws ecr describe-images --repository-name <repo> --image-ids imageTag=<tag>

Read references/ecs-guardrails.md before concluding on any ECS issue.

Step 3 — Detailed path

# ECS Exec for live debugging
aws ecs execute-command --cluster <cluster> --task <task-id> --container <name> --interactive --command "/bin/sh"

# CloudTrail for API events
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=RunTask

# X-Ray traces (if enabled)
aws xray get-trace-summaries --start-time <time> --end-time <time>

Gotchas: ECS

• Fargate tasks get a public IP only if assignPublicIp=ENABLED in awsvpcConfiguration. Without it, they need NAT for internet/ECR access.
• EC2 launch type: container instance must have ECS Agent running, IAM role with AmazonEC2ContainerServiceforEC2Role, and available CPU/memory.
• Task role (taskRoleArn) is for the APPLICATION inside the container. Execution role (executionRoleArn) is for ECS AGENT operations (pulling images, pushing logs, getting secrets).
• OOM kills: if a container exceeds its hard memory limit, ECS kills it (exit code 137). Soft limit (memoryReservation) is for scheduling only.
• Exit code 137 = SIGKILL (OOM or manual stop). Exit code 1 = application error. Exit code 139 = SIGSEGV. Exit code 143 = SIGTERM (graceful stop).
• Image pull failures: check ECR permissions on execution role, VPC endpoint or NAT for ECR access, and image tag existence.
• Service events show the last 100 events. Older events are lost. Check immediately when investigating.
• Deployment circuit breaker rolls back automatically after consecutive failures. Check deployment configuration.
• ECS Exec requires: SSM plugin, task role with ssmmessages permissions, enableExecuteCommand=true on service, and platform version 1.4.0+ (Fargate).
• awsvpc network mode: each task gets its own ENI. Subject to ENI limits per instance (EC2 launch type). Use ENI trunking for higher density.
• Service discovery (Cloud Map): health checks are separate from ELB health checks. Both must pass for traffic to flow.
• Fargate platform versions matter. 1.4.0 added EFS, ephemeral storage config, ECS Exec. Always use LATEST or specific version.

Launch type comparison

Task stopped reasons

Anti-hallucination rules

1. Always cite specific service events, stopped task reasons, or container exit codes as evidence.
2. Task role ≠ execution role. Never confuse which role is used for what.
3. Fargate tasks need NAT or VPC endpoints for internet/ECR access. Never claim Fargate has internet by default.
4. Exit code 137 is OOM or SIGKILL, not always OOM. Check the stoppedReason.
5. awsvpc mode means one ENI per task. ENI limits apply on EC2 launch type.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

38 runbooks