AWS Batch Diagnostics
When to use
Any AWS Batch investigation where the console alone is insufficient — job failures, compute environment issues, scheduling problems, container configuration, networking, array/multi-node jobs, performance optimization, or security troubleshooting.
Investigation workflow
Step 1 — Collect and triage
aws batch describe-compute-environments --compute-environments <ce-name>
aws batch describe-job-queues --job-queues <queue-name>
aws batch describe-jobs --jobs <job-id>
aws batch list-jobs --job-queue <queue-name> --job-status FAILED --max-results 10
aws batch list-jobs --job-queue <queue-name> --job-status RUNNABLE --max-results 10
Step 2 — Domain deep dive
aws batch describe-job-definitions --job-definition-name <def-name> --status ACTIVE
aws batch describe-jobs --jobs <job-id> --query 'jobs[*].{Status:status,Reason:statusReason,Container:container.{ExitCode:exitCode,Reason:reason,LogStream:logStreamName}}'
aws logs get-log-events --log-group-name /aws/batch/job --log-stream-name <stream-name>
aws ec2 describe-instances --filters Name=tag:aws:batch:compute-environment,Values=<ce-name>
Step 3 — Detailed investigation
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=batch.amazonaws.com --max-results 20
aws ecs describe-clusters --clusters <ecs-cluster-arn>
aws cloudwatch get-metric-statistics --namespace AWS/Batch --metric-name CPUUtilization --dimensions Name=JobQueue,Value=<queue-name> --start-time <start> --end-time <end> --period 300 --statistics Average
Read references/batch-guardrails.md before concluding on any AWS Batch issue.
Tool quick reference
| Tool / API |
When to use |
describe-jobs |
Check job status, exit code, reason, log stream |
list-jobs |
List jobs by status in a queue |
describe-compute-environments |
Check CE config, state, instance types |
describe-job-queues |
Check queue priority and CE associations |
describe-job-definitions |
Check container config, resources, IAM |
| CloudWatch Logs |
Check container stdout/stderr logs |
| ECS |
Check underlying ECS cluster and tasks |
| EC2 |
Check instances in managed CE |
Gotchas: AWS Batch
- RUNNABLE status means waiting for resources, not running. Jobs in RUNNABLE are waiting for compute capacity. This can be caused by: CE not scaling, instance types unavailable, VPC/subnet issues, or resource requirements exceeding CE limits. RUNNABLE is the most common stuck state.
- EC2 and Fargate compute environments have different capabilities. EC2 supports GPUs, custom AMIs, placement groups, and more instance types. Fargate is serverless but has vCPU/memory limits and no GPU support. Fargate Spot is available for cost savings.
- Job exit codes indicate failure type. Exit code 0 = success. Non-zero = application failure. Exit code 137 = OOM killed (container exceeded memory). Exit code 1 = general application error. Check container logs for details.
- Spot interruptions can terminate running jobs. Spot instances can be reclaimed with 2-minute warning. Batch retries Spot-interrupted jobs automatically (if retry strategy configured). Use a mix of Spot and On-Demand for critical workloads.
- Array jobs share a single job definition. Each child job gets an AWS_BATCH_JOB_ARRAY_INDEX environment variable. Array size can be 2-10,000. Failed children don't affect other children. Dependencies can be set between array jobs.
- Multi-node parallel jobs require EC2 compute. They use ECS task placement for node coordination. The main node (index 0) coordinates work. All nodes must be able to communicate. Fargate does not support multi-node parallel.
- Job queue priority determines scheduling order. Higher priority queues are evaluated first. Jobs within a queue are FIFO. Fair-share scheduling can override priority-based scheduling. A queue can be associated with multiple CEs.
- Container resource requests must match CE capabilities. vCPU and memory requests must be available in the CE instance types. Requesting more resources than any instance type can provide causes RUNNABLE stuck. GPU requests require GPU instance types.
Anti-hallucination rules
- Always cite specific job IDs, exit codes, or API responses as evidence.
- RUNNABLE means waiting for resources, NOT running. Never confuse the two.
- Exit code 137 means OOM killed. Never ignore memory-related exit codes.
- Fargate does not support GPUs or multi-node parallel. Never suggest these.
- Spot interruptions are expected. Never treat them as unexpected failures.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
24 runbooks
| Category |
IDs |
Covers |
| A — Jobs |
A1-A3 |
Job failures, stuck RUNNABLE, timeout |
| B — Compute |
B1-B3 |
Compute environment failures, EC2 vs Fargate, spot interruptions |
| C — Queue |
C1-C2 |
Job queue issues, scheduling |
| D — Definition |
D1-D2 |
Job definition errors, container config |
| E — Networking |
E1-E2 |
VPC config, ECR access |
| F — Array/Multi |
F1-F2 |
Array jobs, multi-node parallel |
| G — Performance |
G1-G2 |
Throughput, resource optimization |
| H — Security |
H1-H2 |
IAM roles, secrets |
| Z — Catch-All |
Z1 |
General troubleshooting |
1---2name: batch-diagnostics3description: Use this skill to investigate and troubleshoot AWS Batch problems by analyzing job failures, compute environments, job queues, job definitions, networking, array jobs, multi-node parallel jobs, performance, and security using structured runbooks. Activate when: job failures, jobs stuck in RUNNABLE, job timeout, compute environment failures, EC2 vs Fargate issues, spot interruptions, job queue scheduling problems, job definition errors, container configuration issues, VPC networking problems, ECR access failures, array job failures, multi-node parallel issues, throughput bottlenecks, resource optimization, IAM role errors, secrets access problems, or the user says something is wrong with AWS Batch without naming specific symptoms.4---56# AWS Batch Diagnostics78## When to use910Any AWS Batch investigation where the console alone is insufficient — job failures, compute environment issues, scheduling problems, container configuration, networking, array/multi-node jobs, performance optimization, or security troubleshooting.1112## Investigation workflow1314### Step 1 — Collect and triage1516```17aws batch describe-compute-environments --compute-environments <ce-name>18aws batch describe-job-queues --job-queues <queue-name>19aws batch describe-jobs --jobs <job-id>20aws batch list-jobs --job-queue <queue-name> --job-status FAILED --max-results 1021aws batch list-jobs --job-queue <queue-name> --job-status RUNNABLE --max-results 1022```2324### Step 2 — Domain deep dive2526```27aws batch describe-job-definitions --job-definition-name <def-name> --status ACTIVE28aws batch describe-jobs --jobs <job-id> --query 'jobs[*].{Status:status,Reason:statusReason,Container:container.{ExitCode:exitCode,Reason:reason,LogStream:logStreamName}}'29aws logs get-log-events --log-group-name /aws/batch/job --log-stream-name <stream-name>30aws ec2 describe-instances --filters Name=tag:aws:batch:compute-environment,Values=<ce-name>31```3233### Step 3 — Detailed investigation3435```36aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=batch.amazonaws.com --max-results 2037aws ecs describe-clusters --clusters <ecs-cluster-arn>38aws cloudwatch get-metric-statistics --namespace AWS/Batch --metric-name CPUUtilization --dimensions Name=JobQueue,Value=<queue-name> --start-time <start> --end-time <end> --period 300 --statistics Average39```4041Read `references/batch-guardrails.md` before concluding on any AWS Batch issue.4243## Tool quick reference4445| Tool / API | When to use |46|------------|-------------|47| `describe-jobs` | Check job status, exit code, reason, log stream |48| `list-jobs` | List jobs by status in a queue |49| `describe-compute-environments` | Check CE config, state, instance types |50| `describe-job-queues` | Check queue priority and CE associations |51| `describe-job-definitions` | Check container config, resources, IAM |52| CloudWatch Logs | Check container stdout/stderr logs |53| ECS | Check underlying ECS cluster and tasks |54| EC2 | Check instances in managed CE |5556## Gotchas: AWS Batch5758- RUNNABLE status means waiting for resources, not running. Jobs in RUNNABLE are waiting for compute capacity. This can be caused by: CE not scaling, instance types unavailable, VPC/subnet issues, or resource requirements exceeding CE limits. RUNNABLE is the most common stuck state.59- EC2 and Fargate compute environments have different capabilities. EC2 supports GPUs, custom AMIs, placement groups, and more instance types. Fargate is serverless but has vCPU/memory limits and no GPU support. Fargate Spot is available for cost savings.60- Job exit codes indicate failure type. Exit code 0 = success. Non-zero = application failure. Exit code 137 = OOM killed (container exceeded memory). Exit code 1 = general application error. Check container logs for details.61- Spot interruptions can terminate running jobs. Spot instances can be reclaimed with 2-minute warning. Batch retries Spot-interrupted jobs automatically (if retry strategy configured). Use a mix of Spot and On-Demand for critical workloads.62- Array jobs share a single job definition. Each child job gets an AWS_BATCH_JOB_ARRAY_INDEX environment variable. Array size can be 2-10,000. Failed children don't affect other children. Dependencies can be set between array jobs.63- Multi-node parallel jobs require EC2 compute. They use ECS task placement for node coordination. The main node (index 0) coordinates work. All nodes must be able to communicate. Fargate does not support multi-node parallel.64- Job queue priority determines scheduling order. Higher priority queues are evaluated first. Jobs within a queue are FIFO. Fair-share scheduling can override priority-based scheduling. A queue can be associated with multiple CEs.65- Container resource requests must match CE capabilities. vCPU and memory requests must be available in the CE instance types. Requesting more resources than any instance type can provide causes RUNNABLE stuck. GPU requests require GPU instance types.6667## Anti-hallucination rules68691. Always cite specific job IDs, exit codes, or API responses as evidence.702. RUNNABLE means waiting for resources, NOT running. Never confuse the two.713. Exit code 137 means OOM killed. Never ignore memory-related exit codes.724. Fargate does not support GPUs or multi-node parallel. Never suggest these.735. Spot interruptions are expected. Never treat them as unexpected failures.746. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.7576## 24 runbooks7778| Category | IDs | Covers |79|----------|-----|--------|80| A — Jobs | A1-A3 | Job failures, stuck RUNNABLE, timeout |81| B — Compute | B1-B3 | Compute environment failures, EC2 vs Fargate, spot interruptions |82| C — Queue | C1-C2 | Job queue issues, scheduling |83| D — Definition | D1-D2 | Job definition errors, container config |84| E — Networking | E1-E2 | VPC config, ECR access |85| F — Array/Multi | F1-F2 | Array jobs, multi-node parallel |86| G — Performance | G1-G2 | Throughput, resource optimization |87| H — Security | H1-H2 | IAM roles, secrets |88| Z — Catch-All | Z1 | General troubleshooting |
aws-samples