aws-troubleshoot

name	aws-troubleshoot
description	AWS service troubleshooting patterns. Use for EC2, ECS, Lambda, CloudWatch, RDS issues.
allowed-tools	Bash(aws *, python *)

AWS Troubleshooting Expertise

Investigation Methodology

1. Identify the AWS resource/service involved
2. Check resource status using describe functions
3. Review CloudWatch logs for errors
4. Check CloudWatch metrics for anomalies
5. Analyze configuration for misconfigurations
6. Synthesize and recommend

CloudWatch Logs Strategy

Partition First (CRITICAL)

Never dump all logs. Use aggregation queries first:

# Error rate over time
filter @message like /ERROR/
| stats count(*) as errors by bin(5m)

# Top error messages
filter @message like /Exception/
| stats count(*) by @message
| sort count desc
| limit 10

# Latency percentiles
stats pct(@duration, 50) as p50, pct(@duration, 99) as p99 by bin(5m)

# Unique error types
filter @message like /ERROR/
| parse @message /(?<error_type>[\w.]+Exception)/
| stats count(*) by error_type

Query Flow

1. Statistics first: Get error counts, distributions
2. Identify time windows: Find when errors spiked
3. Sample from spikes: Get specific examples
4. Compare to baseline: Query same period yesterday/last week

Service-Specific Patterns

EC2 Issues

Symptom	First Check	Typical Cause
Unreachable	describe_ec2_instance	Security group, stopped, status check failed
Performance	get_cloudwatch_metrics (CPUUtilization)	CPU exhaustion, network saturation
Disk full	get_cloudwatch_metrics (DiskSpaceUtilization)	Logs, temp files

Key CloudWatch metrics for EC2:

• CPUUtilization
• NetworkIn, NetworkOut
• DiskReadOps, DiskWriteOps
• StatusCheckFailed

Lambda Issues

Symptom	First Check	Typical Cause
Timeout	CloudWatch logs	External call slow, cold start, insufficient memory
Permission denied	CloudWatch logs	IAM role missing permissions
Memory error	CloudWatch metrics	Memory allocation too low
Cold starts	CloudWatch logs + metrics	Provisioned concurrency needed

Key CloudWatch metrics for Lambda:

• Invocations
• Duration
• Errors
• Throttles
• ConcurrentExecutions

CloudWatch Insights for Lambda:

# Cold start analysis
filter @type = "REPORT"
| stats avg(@initDuration) as avg_cold_start,
        count(@initDuration) as cold_starts,
        count(*) as total_invocations
        by bin(5m)

# Timeout analysis
filter @message like /Task timed out/
| stats count(*) by bin(5m)

ECS/Fargate Issues

Symptom	First Check	Typical Cause
Task failed	list_ecs_tasks	Container crash, resource limits, image pull
Service unhealthy	list_ecs_tasks	Health check failing, target group issues
Slow scaling	CloudWatch metrics	Insufficient capacity, service limits

Investigation flow:

1. list_ecs_tasks - See task status and health
2. Check stopped reason in task description
3. Review CloudWatch logs for the task
4. Check container insights metrics

RDS Issues

Symptom	First Check	Typical Cause
Connection refused	get_rds_instance_status	Security group, stopped, maintenance
Slow queries	CloudWatch metrics	CPU, IOPS, connections
Storage full	CloudWatch metrics	Data growth, logs, snapshots

Key CloudWatch metrics for RDS:

• CPUUtilization
• DatabaseConnections
• ReadIOPS, WriteIOPS
• FreeStorageSpace
• ReadLatency, WriteLatency

Common AWS Errors

Permission Errors

AccessDeniedException
UnauthorizedAccess

→ Check IAM role/policy attached to the service

Throttling

Throttling
Rate exceeded
TooManyRequestsException

→ Implement exponential backoff, request limit increase

Resource Not Found

ResourceNotFoundException
NoSuchEntity

→ Verify resource name, region, account

Practical AWS CLI Commands

EC2

aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query 'Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name,Name:Tags[?Key==`Name`].Value|[0]}'

aws ec2 describe-instance-status --instance-ids <id>

ECS

aws ecs list-clusters

aws ecs list-services --cluster <cluster>

aws ecs describe-services --cluster <cluster> --services <service>

aws ecs list-tasks --cluster <cluster> --service-name <service> --desired-status STOPPED

CloudWatch Logs

aws logs describe-log-groups --log-group-name-prefix /ecs/

aws logs filter-log-events --log-group-name <group> --start-time <epoch-ms> --filter-pattern "ERROR"

aws logs start-query --log-group-name <group> --start-time <epoch> --end-time <epoch> --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50'

EKS

aws eks list-clusters

aws eks describe-cluster --name <cluster>

aws eks update-kubeconfig --name <cluster> --region <region>

RDS

aws rds describe-db-instances --query 'DBInstances[].{ID:DBInstanceIdentifier,Engine:Engine,Status:DBInstanceStatus,Class:DBInstanceClass}'

aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name CPUUtilization --dimensions Name=DBInstanceIdentifier,Value=<id> --start-time <iso> --end-time <iso> --period 300 --statistics Average