# AWS Troubleshoot

> AWS service troubleshooting patterns. Use for EC2, ECS, Lambda, CloudWatch, RDS issues.

- Skill: `incidentfox/aws-troubleshoot-2` (Agent Skill)
- Install (CLI): `npx skillmds@latest add incidentfox/aws-troubleshoot-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/incidentfox/aws-troubleshoot-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: incidentfox (https://skillmd.com/u/incidentfox)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/incidentfox/aws-troubleshoot-2

---


# AWS Troubleshooting Expertise

## Investigation Methodology

1. **Identify the AWS resource/service involved**
2. **Check resource status** using describe functions
3. **Review CloudWatch logs** for errors
4. **Check CloudWatch metrics** for anomalies
5. **Analyze configuration** for misconfigurations
6. **Synthesize and recommend**

## CloudWatch Logs Strategy

### Partition First (CRITICAL)

**Never dump all logs.** Use aggregation queries first:

```
# Error rate over time
filter @message like /ERROR/
| stats count(*) as errors by bin(5m)

# Top error messages
filter @message like /Exception/
| stats count(*) by @message
| sort count desc
| limit 10

# Latency percentiles
stats pct(@duration, 50) as p50, pct(@duration, 99) as p99 by bin(5m)

# Unique error types
filter @message like /ERROR/
| parse @message /(?<error_type>[\w.]+Exception)/
| stats count(*) by error_type
```

### Query Flow

1. **Statistics first**: Get error counts, distributions
2. **Identify time windows**: Find when errors spiked
3. **Sample from spikes**: Get specific examples
4. **Compare to baseline**: Query same period yesterday/last week

## Service-Specific Patterns

### EC2 Issues

| Symptom | First Check | Typical Cause |
|---------|-------------|---------------|
| Unreachable | `describe_ec2_instance` | Security group, stopped, status check failed |
| Performance | `get_cloudwatch_metrics` (CPUUtilization) | CPU exhaustion, network saturation |
| Disk full | `get_cloudwatch_metrics` (DiskSpaceUtilization) | Logs, temp files |

**Key CloudWatch metrics for EC2**:
- CPUUtilization
- NetworkIn, NetworkOut
- DiskReadOps, DiskWriteOps
- StatusCheckFailed

### Lambda Issues

| Symptom | First Check | Typical Cause |
|---------|-------------|---------------|
| Timeout | CloudWatch logs | External call slow, cold start, insufficient memory |
| Permission denied | CloudWatch logs | IAM role missing permissions |
| Memory error | CloudWatch metrics | Memory allocation too low |
| Cold starts | CloudWatch logs + metrics | Provisioned concurrency needed |

**Key CloudWatch metrics for Lambda**:
- Invocations
- Duration
- Errors
- Throttles
- ConcurrentExecutions

**CloudWatch Insights for Lambda**:
```
# Cold start analysis
filter @type = "REPORT"
| stats avg(@initDuration) as avg_cold_start,
        count(@initDuration) as cold_starts,
        count(*) as total_invocations
        by bin(5m)

# Timeout analysis
filter @message like /Task timed out/
| stats count(*) by bin(5m)
```

### ECS/Fargate Issues

| Symptom | First Check | Typical Cause |
|---------|-------------|---------------|
| Task failed | `list_ecs_tasks` | Container crash, resource limits, image pull |
| Service unhealthy | `list_ecs_tasks` | Health check failing, target group issues |
| Slow scaling | CloudWatch metrics | Insufficient capacity, service limits |

**Investigation flow**:
1. `list_ecs_tasks` - See task status and health
2. Check stopped reason in task description
3. Review CloudWatch logs for the task
4. Check container insights metrics

### RDS Issues

| Symptom | First Check | Typical Cause |
|---------|-------------|---------------|
| Connection refused | `get_rds_instance_status` | Security group, stopped, maintenance |
| Slow queries | CloudWatch metrics | CPU, IOPS, connections |
| Storage full | CloudWatch metrics | Data growth, logs, snapshots |

**Key CloudWatch metrics for RDS**:
- CPUUtilization
- DatabaseConnections
- ReadIOPS, WriteIOPS
- FreeStorageSpace
- ReadLatency, WriteLatency

## Common AWS Errors

### Permission Errors
```
AccessDeniedException
UnauthorizedAccess
```
→ Check IAM role/policy attached to the service

### Throttling
```
Throttling
Rate exceeded
TooManyRequestsException
```
→ Implement exponential backoff, request limit increase

### Resource Not Found
```
ResourceNotFoundException
NoSuchEntity
```
→ Verify resource name, region, account

## Practical AWS CLI Commands

### EC2
```bash
aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query 'Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name,Name:Tags[?Key==`Name`].Value|[0]}'
```
```bash
aws ec2 describe-instance-status --instance-ids <id>
```

### ECS
```bash
aws ecs list-clusters
```
```bash
aws ecs list-services --cluster <cluster>
```
```bash
aws ecs describe-services --cluster <cluster> --services <service>
```
```bash
aws ecs list-tasks --cluster <cluster> --service-name <service> --desired-status STOPPED
```

### CloudWatch Logs
```bash
aws logs describe-log-groups --log-group-name-prefix /ecs/
```
```bash
aws logs filter-log-events --log-group-name <group> --start-time <epoch-ms> --filter-pattern "ERROR"
```
```bash
aws logs start-query --log-group-name <group> --start-time <epoch> --end-time <epoch> --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50'
```

### EKS
```bash
aws eks list-clusters
```
```bash
aws eks describe-cluster --name <cluster>
```
```bash
aws eks update-kubeconfig --name <cluster> --region <region>
```

### RDS
```bash
aws rds describe-db-instances --query 'DBInstances[].{ID:DBInstanceIdentifier,Engine:Engine,Status:DBInstanceStatus,Class:DBInstanceClass}'
```
```bash
aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name CPUUtilization --dimensions Name=DBInstanceIdentifier,Value=<id> --start-time <iso> --end-time <iso> --period 300 --statistics Average
```