| name | elasticache-diagnostics |
| version | 1.0.0 |
| last_updated | 2025-04-12 |
| description | Use this skill to investigate and troubleshoot Amazon ElastiCache problems (both Redis and Memcached engines) by analyzing cluster configurations, replication groups, parameter groups, CloudWatch metrics, slow logs, and following structured runbooks. Activate when: connection failures, timeout issues, DNS/endpoint resolution, high CPU utilization, memory pressure or evictions, high latency, slow commands, replication lag, failover issues, read replica problems, vertical or horizontal scaling, ElastiCache Serverless issues, eviction storms, memory fragmentation, maxmemory configuration, AUTH token issues, encryption problems, security group/subnet issues, backup failures, restore issues, slot migration, resharding failures, cross-slot errors, Memcached auto-discovery, consistent hashing, Global Datastore replication or failover, or the user says something is wrong with ElastiCache without naming specific symptoms.
|
| compatibility | Requires AWS CLI or SDK access with ElastiCache, CloudWatch, CloudTrail, EC2 (for security groups/subnets), KMS, and optionally SNS permissions. Redis CLI (redis-cli) recommended for direct node diagnostics.
|
ElastiCache Diagnostics
When to use
Any ElastiCache investigation where the console alone is insufficient — connection debugging, performance analysis, replication troubleshooting, scaling operations, memory management, security configuration, backup/restore, cluster mode operations, Memcached-specific issues, or Global Datastore problems.
Investigation workflow
Step 1 — Collect and triage
aws elasticache describe-cache-clusters --show-cache-node-info
aws elasticache describe-replication-groups
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name CPUUtilization --dimensions Name=CacheClusterId,Value=<cluster-id> --start-time <start> --end-time <end> --period 300 --statistics Average
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name EngineCPUUtilization --dimensions Name=CacheClusterId,Value=<cluster-id> --start-time <start> --end-time <end> --period 300 --statistics Average
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name DatabaseMemoryUsagePercentage --dimensions Name=CacheClusterId,Value=<cluster-id> --start-time <start> --end-time <end> --period 300 --statistics Average
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name Evictions --dimensions Name=CacheClusterId,Value=<cluster-id> --start-time <start> --end-time <end> --period 300 --statistics Sum
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name CurrConnections --dimensions Name=CacheClusterId,Value=<cluster-id> --start-time <start> --end-time <end> --period 300 --statistics Maximum
Step 2 — Domain deep dive
aws elasticache describe-events --source-type cache-cluster --duration 1440
aws elasticache describe-events --source-type replication-group --duration 1440
aws elasticache describe-cache-parameters --cache-parameter-group-name <param-group>
redis-cli -h <endpoint> -p 6379 INFO
redis-cli -h <endpoint> -p 6379 SLOWLOG GET 25
redis-cli -h <endpoint> -p 6379 INFO replication
redis-cli -h <endpoint> -p 6379 INFO memory
redis-cli -h <endpoint> -p 6379 CLIENT LIST
Step 3 — Detailed investigation
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=elasticache.amazonaws.com --max-results 20
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name ReplicationLag --dimensions Name=CacheClusterId,Value=<replica-id> --start-time <start> --end-time <end> --period 60 --statistics Maximum
aws elasticache describe-cache-subnet-groups --cache-subnet-group-name <subnet-group>
aws ec2 describe-security-groups --group-ids <sg-id>
aws elasticache describe-snapshots --replication-group-id <repl-group-id>
aws elasticache describe-global-replication-groups
Read references/elasticache-guardrails.md before concluding on any ElastiCache issue.
Tool quick reference
| Tool / API | When to use |
|---|
describe-cache-clusters | Full cluster details and node info |
describe-replication-groups | Redis replication topology and endpoints |
describe-events | Recent cluster and replication group events |
describe-cache-parameters | Parameter group settings |
describe-cache-subnet-groups | Subnet group and VPC configuration |
describe-snapshots | Backup/snapshot details (Redis only) |
describe-global-replication-groups | Global Datastore configuration |
describe-cache-engine-versions | Available engine versions |
describe-service-updates | Pending service updates |
redis-cli INFO | Real-time Redis server stats |
redis-cli SLOWLOG | Slow command analysis |
redis-cli CLIENT LIST | Active connection details |
Gotchas: ElastiCache
- Cluster mode enabled vs disabled (Redis): Cluster mode enabled (CME) distributes data across multiple shards using hash slots (0-16383). Cluster mode disabled (CMD) has a single shard with one primary and up to 5 replicas. CME supports up to 500 nodes. CMD cannot be converted to CME without creating a new cluster. CME requires the Redis Cluster client.
- Replication group vs cache cluster terminology: A replication group is the Redis logical grouping (primary + replicas). Each node in the replication group is a cache cluster. The replication group endpoint is the primary endpoint. Do not confuse cache cluster IDs with replication group IDs.
- Failover behavior (Multi-AZ): Multi-AZ with automatic failover promotes a read replica to primary when the primary fails. DNS propagation takes 30-60 seconds. Applications using the primary endpoint will automatically connect to the new primary. Failover can cause brief write unavailability (typically under 60 seconds).
- Eviction policies: Redis supports allkeys-lru, volatile-lru, allkeys-lfu, volatile-lfu, allkeys-random, volatile-random, volatile-ttl, and noeviction. Default is volatile-lru. volatile-* policies only evict keys with TTL set. If no keys have TTL and policy is volatile-*, Redis returns OOM errors. allkeys-lru is recommended for cache-only workloads.
- maxmemory-reserved: AWS recommends reserving 25% of maxmemory for non-data overhead (replication buffer, Lua scripts, connections, pub/sub). Default is set per node type. Insufficient reservation causes OOM during replication or heavy write loads. This is set in the parameter group, not in Redis config directly.
- Connection limits per node type: Each node type has a maximum connection limit (maxclients). cache.t3.micro supports ~65,000 connections. Larger node types support more. Monitor CurrConnections in CloudWatch. Connection storms can exhaust limits and cause new connection failures.
- Redis AUTH token rotation: AUTH tokens can be rotated using modify-replication-group with --auth-token and --auth-token-update-strategy (SET or ROTATE). ROTATE allows both old and new tokens during transition. SET immediately replaces the token. Use ROTATE for zero-downtime rotation.
- Redis 7.x vs 6.x feature differences: Redis 7.x adds Redis Functions (server-side scripting), ACL improvements, sharded pub/sub, multi-part AOF, and listpack encoding. Redis 6.x introduced ACLs, client-side caching, and RESP3. Not all Redis OSS features are available in ElastiCache — check AWS documentation.
- Memcached auto-discovery: ElastiCache Memcached supports auto-discovery — clients automatically detect nodes added or removed. Requires the ElastiCache Cluster Client (not standard memcached clients). The configuration endpoint returns the current node list. Standard clients must be manually updated when nodes change.
- ElastiCache Serverless vs provisioned: Serverless automatically scales compute and memory. No node type selection needed. Pricing is based on data stored (per GB-hour) and ElastiCache Processing Units (ECPUs). Serverless has different limits than provisioned (e.g., max item size, connection limits). Not all Redis commands are supported in Serverless.
Anti-hallucination rules
- Always cite specific cluster configurations, parameter group values, CloudWatch metrics, or redis-cli output as evidence.
- Never confuse replication group IDs with cache cluster IDs. A replication group contains one or more cache clusters (nodes).
- Memcached does NOT support replication, persistence, backups, or failover. Never suggest these features for Memcached clusters.
- In-transit encryption (TLS) can only be enabled at cluster creation time. Never suggest enabling TLS on an existing non-TLS cluster without recreation.
- ElastiCache clusters are NOT publicly accessible. Never suggest connecting directly from the internet — always require VPC access (bastion, VPN, or VPC peering).
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
32 runbooks
| Category | IDs | Covers |
|---|
| A — Connectivity | A1-A3 | Connection failures, timeout issues, DNS/endpoint resolution |
| B — Performance | B1-B4 | High CPU, memory pressure/evictions, high latency, slow commands |
| C — Replication | C1-C3 | Replication lag, failover issues, read replica problems |
| D — Scaling | D1-D3 | Vertical scaling, horizontal scaling (resharding), ElastiCache Serverless |
| E — Memory | E1-E3 | Eviction storms, memory fragmentation, maxmemory configuration |
| F — Security | F1-F3 | AUTH token issues, encryption, security group/subnet |
| G — Backup | G1-G2 | Backup failures, restore issues |
| H — Cluster Mode | H1-H3 | Slot migration, resharding failures, cross-slot errors |
| I — Memcached | I1-I2 | Auto-discovery, consistent hashing |
| J — Global Datastore | J1-J2 | Replication lag, failover |
| Z — Catch-All | Z1 | General troubleshooting |
