Mwaa Diagnostics

Amazon MWAA Diagnostics

When to use

Any Amazon MWAA investigation where the console alone is insufficient — environment health analysis, DAG import debugging, task failure troubleshooting, scheduling issues, VPC networking, web server access, worker scaling, plugin/requirements problems, logging issues, IAM permissions, secrets backend, or S3 bucket sync.

Investigation workflow

Step 1 — Collect and triage

aws mwaa list-environments
aws mwaa get-environment --name <env-name>
aws mwaa get-environment --name <env-name> --query 'Environment.Status'
aws mwaa get-environment --name <env-name> --query 'Environment.{Status:Status,LastUpdate:LastUpdate}'
aws cloudwatch get-metric-statistics --namespace AWS/MWAA --metric-name RunningTasks --dimensions Name=Environment,Value=<env-name> --start-time <start> --end-time <end> --period 300 --statistics Average
aws logs describe-log-groups --log-group-name-prefix airflow-<env-name>

Step 2 — Domain deep dive

aws mwaa get-environment --name <env-name> --query 'Environment.{DagS3Path:DagS3Path,SourceBucketArn:SourceBucketArn,PluginsS3Path:PluginsS3Path,RequirementsS3Path:RequirementsS3Path}'
aws mwaa get-environment --name <env-name> --query 'Environment.ExecutionRoleArn'
aws iam get-role --role-name <execution-role-name>
aws iam list-attached-role-policies --role-name <execution-role-name>
aws s3 ls s3://<bucket-name>/<dags-path>/
aws mwaa create-cli-token --name <env-name>

Step 3 — Detailed investigation

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=airflow.amazonaws.com --max-results 20
aws logs filter-log-events --log-group-name airflow-<env-name>-DAGProcessing --start-time <epoch-ms> --filter-pattern "ERROR"
aws logs filter-log-events --log-group-name airflow-<env-name>-Task --start-time <epoch-ms> --filter-pattern "ERROR"
aws ec2 describe-security-groups --group-ids <sg-id>
aws ec2 describe-subnets --subnet-ids <subnet-ids>
aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=<vpc-id>

Read references/guardrails.md before concluding on any MWAA issue.

Tool quick reference

Gotchas: Amazon MWAA

• MWAA is managed Airflow — you cannot SSH into workers, schedulers, or web servers. No access to airflow.cfg directly. Configuration is via environment variables with the AIRFLOW__ prefix or through the MWAA console/API.
• DAGs must be in S3 — MWAA syncs DAGs from an S3 bucket. DAG parse errors happen during sync, not at deploy time. Check the DAGProcessing log group for import errors.
• requirements.txt runs at environment startup — if a package fails to install, the entire environment can become unhealthy. Always test requirements locally first. Version pins are critical.
• plugins.zip is extracted to the Airflow plugins directory — custom operators, hooks, and sensors go here. The ZIP structure must match Airflow's expected plugin layout.
• VPC requires private subnets with NAT gateway — MWAA environments need private subnets with outbound internet access (via NAT) for PyPI package installation and AWS API calls. No public subnet support.
• Web server access modes: PUBLIC_ONLY or PRIVATE_ONLY — public mode creates an internet-facing endpoint; private mode requires VPC endpoint access. Switching modes requires environment update.
• Execution role is the single IAM identity for all Airflow operations — S3 DAG access, CloudWatch logging, Secrets Manager, KMS, and any AWS services called from DAGs all use this role.
• Airflow version upgrades are in-place but require careful testing — not all DAGs are compatible across Airflow versions. Test DAGs against the target version before upgrading.
• Worker auto-scaling is based on task queue depth — min/max workers are configurable. Scaling takes minutes, not seconds. Tasks may queue during scale-up.
• Secrets backend (Secrets Manager or Parameter Store) requires specific naming conventions — connection IDs and variables must follow the configured prefix pattern.
• CloudWatch log groups are per-component — separate log groups for Scheduler, Worker, WebServer, DAGProcessing, and Task logs. Each can be enabled/disabled independently.
• Environment updates can take 20-30 minutes — updates are not instant. The environment enters UPDATING status and may be partially unavailable during the update.

Anti-hallucination rules

1. Always cite specific environment configurations, IAM policies, log entries, or API responses as evidence. Never guess at environment settings.
2. MWAA does NOT allow SSH access to any component. Never suggest connecting to workers or schedulers directly.
3. DAGs must be in S3 — never suggest placing DAGs on local file systems or using git-sync directly.
4. requirements.txt failures can break the entire environment. Never dismiss package installation errors as non-critical.
5. VPC must use private subnets with NAT. Never suggest public subnets for MWAA.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

24 runbooks