| name | mwaa-diagnostics |
| version | 1.0.0 |
| last_updated | 2025-04-12 |
| description | Use this skill to investigate and troubleshoot Amazon Managed Workflows for Apache Airflow (MWAA) problems by analyzing environment configurations, DAG import errors, task failures, scheduling issues, VPC networking, web server access, worker scaling, plugin and requirements issues, CloudWatch logging, IAM execution roles, secrets backend, S3 DAG bucket sync, and Airflow connections using structured runbooks. Activate when: environment creation or update failures, version upgrade issues, DAG import errors, task execution failures, scheduling problems, VPC networking issues, web server inaccessible, worker auto-scaling problems, task queue backlog, custom plugin failures, requirements.txt installation errors, CloudWatch log delivery issues, task log missing, IAM execution role permission errors, secrets backend configuration problems, S3 DAG bucket sync failures, Airflow connection issues, or the user says something is wrong with MWAA without naming specific symptoms.
|
| compatibility | Requires AWS CLI or SDK access with MWAA, S3, IAM, CloudWatch Logs, EC2 (VPC), CloudTrail, SecretsManager, and optionally KMS permissions. Airflow CLI access via MWAA CLI token for DAG and task operations.
|
Amazon MWAA Diagnostics
When to use
Any Amazon MWAA investigation where the console alone is insufficient — environment health analysis, DAG import debugging, task failure troubleshooting, scheduling issues, VPC networking, web server access, worker scaling, plugin/requirements problems, logging issues, IAM permissions, secrets backend, or S3 bucket sync.
Investigation workflow
Step 1 — Collect and triage
aws mwaa list-environments
aws mwaa get-environment --name <env-name>
aws mwaa get-environment --name <env-name> --query 'Environment.Status'
aws mwaa get-environment --name <env-name> --query 'Environment.{Status:Status,LastUpdate:LastUpdate}'
aws cloudwatch get-metric-statistics --namespace AWS/MWAA --metric-name RunningTasks --dimensions Name=Environment,Value=<env-name> --start-time <start> --end-time <end> --period 300 --statistics Average
aws logs describe-log-groups --log-group-name-prefix airflow-<env-name>
Step 2 — Domain deep dive
aws mwaa get-environment --name <env-name> --query 'Environment.{DagS3Path:DagS3Path,SourceBucketArn:SourceBucketArn,PluginsS3Path:PluginsS3Path,RequirementsS3Path:RequirementsS3Path}'
aws mwaa get-environment --name <env-name> --query 'Environment.ExecutionRoleArn'
aws iam get-role --role-name <execution-role-name>
aws iam list-attached-role-policies --role-name <execution-role-name>
aws s3 ls s3://<bucket-name>/<dags-path>/
aws mwaa create-cli-token --name <env-name>
Step 3 — Detailed investigation
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=airflow.amazonaws.com --max-results 20
aws logs filter-log-events --log-group-name airflow-<env-name>-DAGProcessing --start-time <epoch-ms> --filter-pattern "ERROR"
aws logs filter-log-events --log-group-name airflow-<env-name>-Task --start-time <epoch-ms> --filter-pattern "ERROR"
aws ec2 describe-security-groups --group-ids <sg-id>
aws ec2 describe-subnets --subnet-ids <subnet-ids>
aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=<vpc-id>
Read references/guardrails.md before concluding on any MWAA issue.
Tool quick reference
| Tool / API | When to use |
|---|
mwaa list-environments | List all MWAA environments |
mwaa get-environment | Full environment details, status, config |
mwaa create-cli-token | Get CLI token for Airflow CLI commands |
s3 ls | Check DAG files in S3 bucket |
logs filter-log-events | Search scheduler, worker, DAG processing logs |
iam get-role | Execution role details |
iam list-attached-role-policies | Policies on execution role |
cloudtrail lookup-events | Recent MWAA API events |
ec2 describe-security-groups | VPC security group rules |
ec2 describe-nat-gateways | NAT gateway for internet access |
secretsmanager list-secrets | Secrets backend entries |
Airflow CLI (via token) | DAG list, task state, trigger, connections |
Gotchas: Amazon MWAA
- MWAA is managed Airflow — you cannot SSH into workers, schedulers, or web servers. No access to airflow.cfg directly. Configuration is via environment variables with the
AIRFLOW__ prefix or through the MWAA console/API.
- DAGs must be in S3 — MWAA syncs DAGs from an S3 bucket. DAG parse errors happen during sync, not at deploy time. Check the DAGProcessing log group for import errors.
- requirements.txt runs at environment startup — if a package fails to install, the entire environment can become unhealthy. Always test requirements locally first. Version pins are critical.
- plugins.zip is extracted to the Airflow plugins directory — custom operators, hooks, and sensors go here. The ZIP structure must match Airflow's expected plugin layout.
- VPC requires private subnets with NAT gateway — MWAA environments need private subnets with outbound internet access (via NAT) for PyPI package installation and AWS API calls. No public subnet support.
- Web server access modes: PUBLIC_ONLY or PRIVATE_ONLY — public mode creates an internet-facing endpoint; private mode requires VPC endpoint access. Switching modes requires environment update.
- Execution role is the single IAM identity for all Airflow operations — S3 DAG access, CloudWatch logging, Secrets Manager, KMS, and any AWS services called from DAGs all use this role.
- Airflow version upgrades are in-place but require careful testing — not all DAGs are compatible across Airflow versions. Test DAGs against the target version before upgrading.
- Worker auto-scaling is based on task queue depth — min/max workers are configurable. Scaling takes minutes, not seconds. Tasks may queue during scale-up.
- Secrets backend (Secrets Manager or Parameter Store) requires specific naming conventions — connection IDs and variables must follow the configured prefix pattern.
- CloudWatch log groups are per-component — separate log groups for Scheduler, Worker, WebServer, DAGProcessing, and Task logs. Each can be enabled/disabled independently.
- Environment updates can take 20-30 minutes — updates are not instant. The environment enters UPDATING status and may be partially unavailable during the update.
Anti-hallucination rules
- Always cite specific environment configurations, IAM policies, log entries, or API responses as evidence. Never guess at environment settings.
- MWAA does NOT allow SSH access to any component. Never suggest connecting to workers or schedulers directly.
- DAGs must be in S3 — never suggest placing DAGs on local file systems or using git-sync directly.
- requirements.txt failures can break the entire environment. Never dismiss package installation errors as non-critical.
- VPC must use private subnets with NAT. Never suggest public subnets for MWAA.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
24 runbooks
| Category | IDs | Covers |
|---|
| A — Environment | A1-A3 | Creation failures, update failures, version upgrades |
| B — DAGs | B1-B3 | DAG import errors, task failures, scheduling issues |
| C — Connectivity | C1-C2 | VPC networking, web server access |
| D — Performance | D1-D2 | Worker scaling, task queue backlog |
| E — Plugins | E1-E2 | Custom plugins, requirements.txt |
| F — Logging | F1-F2 | CloudWatch logs, task logs |
| G — Security | G1-G2 | IAM execution role, secrets backend |
| H — Integration | H1-H2 | S3 DAG bucket, connections |
| Z — Catch-All | Z1 | General troubleshooting |
