| name | emr-diagnostics |
| version | 1.0.0 |
| last_updated | 2025-04-12 |
| description | Use this skill to investigate and troubleshoot Amazon EMR problems by analyzing cluster launch failures, termination issues, bootstrap actions, step failures, Spark tuning, YARN resource management, shuffle issues, EMRFS/S3 issues, HDFS problems, VPC configuration, security groups, managed scaling, instance fleets, EMR on EKS virtual clusters, job run failures, EMR Serverless application failures, capacity issues, Kerberos authentication, Lake Formation integration, EMR Studio, Jupyter issues, and following structured runbooks. Activate when: cluster won't launch, unexpected termination, bootstrap action failures, step execution errors, Spark performance issues, YARN resource contention, shuffle failures, S3 read/write errors, HDFS capacity issues, VPC connectivity problems, scaling not working, EMR on EKS failures, Serverless job errors, security configuration issues, or the user says something is wrong with EMR.
|
| compatibility | Requires AWS CLI or SDK access with EMR, EC2, S3, IAM, CloudWatch, CloudTrail, and optionally EKS, Lake Formation, and VPC permissions. Some operations require EMR cluster access via SSH or EMR Studio.
|
Amazon EMR Diagnostics
When to use
Any Amazon EMR investigation where the console alone is insufficient — cluster lifecycle issues, application performance tuning, storage problems, networking configuration, scaling behavior, EMR on EKS, EMR Serverless, security configuration, or notebook/studio access.
Investigation workflow
Step 1 — Collect and triage
aws emr describe-cluster --cluster-id <cluster-id> --query 'Cluster.{State:Status.State,StateChangeReason:Status.StateChangeReason,Timeline:Status.Timeline}'
aws emr list-clusters --active --query 'Clusters[*].{Id:Id,Name:Name,State:Status.State}'
version: "1.0.0"
last_updated: "2025-04-12"
aws emr list-steps --cluster-id <cluster-id> --query 'Steps[*].{Id:Id,Name:Name,State:Status.State}'
version: "1.0.0"
last_updated: "2025-04-12"
aws emr list-bootstrap-actions --cluster-id <cluster-id>
Step 2 — Domain deep dive
aws emr describe-cluster --cluster-id <cluster-id> --query 'Cluster.{Ec2:Ec2InstanceAttributes,Apps:Applications,Config:Configurations}'
aws emr list-instance-groups --cluster-id <cluster-id> --query 'InstanceGroups[*].{Type:InstanceGroupType,InstanceType:InstanceType,Running:RunningInstanceCount,Requested:RequestedInstanceCount}'
aws emr describe-step --cluster-id <cluster-id> --step-id <step-id>
aws cloudwatch get-metric-statistics --namespace AWS/ElasticMapReduce --metric-name IsIdle --dimensions Name=JobFlowId,Value=<cluster-id> --start-time <start> --end-time <end> --period 300 --statistics Average
Step 3 — Detailed investigation
aws s3 ls s3://<log-bucket>/<cluster-id>/
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=elasticmapreduce.amazonaws.com --max-results 20
aws emr describe-cluster --cluster-id <cluster-id> --query 'Cluster.{SecurityConfig:SecurityConfiguration,ServiceRole:ServiceRole,AutoScaling:AutoScalingRole}'
Read references/emr-guardrails.md before concluding on any EMR issue.
Tool quick reference
| Tool / API | When to use |
|---|
emr describe-cluster | Check cluster status, configuration, and state change reason |
emr list-steps | Check step execution status |
emr list-bootstrap-actions | Check bootstrap action configuration |
emr list-instance-groups | Check instance group status and capacity |
emr describe-step | Get detailed step execution information |
emr list-clusters | List active/terminated clusters |
| S3 log bucket | Access cluster logs, step logs, application logs |
| CloudWatch metrics | Monitor cluster performance metrics |
| YARN ResourceManager UI | Check application status and resource allocation |
Gotchas: Amazon EMR
- EMR cluster logs are in S3, not CloudWatch by default. Logs are written to the configured S3 log bucket. Check s3://// for step logs, application logs, and node logs. Enable CloudWatch logging separately if needed.
- Bootstrap actions run BEFORE applications start. If a bootstrap action fails, the cluster terminates. Bootstrap actions have a timeout. They run on all nodes (master, core, task). Debug by checking bootstrap action logs in S3.
- YARN resource allocation determines application performance. Spark executors, memory, and cores are controlled by YARN. Misconfigured YARN settings lead to underutilization or OOM errors. Check yarn-site.xml and spark-defaults.conf.
- EMRFS consistent view is deprecated. Use S3 strong consistency (default since Dec 2020). Do not configure EMRFS consistent view for new clusters. S3 provides strong read-after-write consistency natively.
- Managed scaling adjusts capacity based on workload. It adds/removes instances automatically. Scaling has a cooldown period. Minimum and maximum units must be configured. Scaling decisions are based on YARN metrics.
- EMR on EKS runs Spark on existing EKS clusters. It requires a virtual cluster, execution role, and job template. Debugging requires checking both EMR and EKS logs. Pod scheduling issues are common.
- EMR Serverless automatically provisions resources. No cluster management needed. Applications have pre-initialized capacity settings. Job failures often relate to IAM permissions or resource limits.
- Security configurations are immutable after cluster creation. Kerberos, encryption, and Lake Formation settings cannot be changed on a running cluster. Create a new cluster with the correct security configuration.
- Instance fleets provide flexibility in instance selection. EMR chooses from a list of instance types based on availability and price. On-Demand and Spot instances can be mixed. Spot interruptions can affect running tasks.
Anti-hallucination rules
- Always cite specific cluster IDs, step IDs, or log entries as evidence.
- EMRFS consistent view is deprecated. Never recommend configuring it.
- Bootstrap actions run before applications. Never suggest modifying running applications via bootstrap.
- Security configurations are immutable. Never suggest changing security config on a running cluster.
- S3 has strong consistency natively. Never suggest EMRFS workarounds for consistency.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
28 runbooks
| Category | IDs | Covers |
|---|
| A — Cluster | A1-A4 | Launch failures, termination, bootstrap actions, step failures |
| B — Performance | B1-B3 | Spark tuning, YARN resource management, shuffle issues |
| C — Storage | C1-C2 | EMRFS/S3 issues, HDFS |
| D — Networking | D1-D2 | VPC config, security groups |
| E — Scaling | E1-E2 | Managed scaling, instance fleet |
| F — EMR on EKS | F1-F2 | Virtual cluster, job run failures |
| G — EMR Serverless | G1-G2 | Application failures, capacity |
| H — Security | H1-H2 | Kerberos, Lake Formation integration |
| I — Notebooks | I1-I2 | EMR Studio, Jupyter issues |
| Z — Catch-All | Z1 | General troubleshooting |
