| name | redis-cloud-failover-test |
| description | Plan, schedule, and validate Redis Cloud failover tests. Use when the user asks to run a Redis Cloud shard failover, node failure test, disaster recovery drill, go-live HA test, endpoint rebind test, RTO/RPO validation, client reconnect testing, DNS TTL troubleshooting, support-scheduled failover, or Pro-plan failover test with Redis Support and customer chat-room presence. |
Redis Cloud Failover Test
Use this skill when a user needs to validate application behavior during Redis Cloud failover or disaster recovery.
Safety Rules
- Redis Cloud failover tests are coordinated with Redis Support; do not present them as arbitrary self-serve commands.
- Prefer staging or QA before production.
- Confirm HA replication before shard or node failover tests.
- Confirm persistence/backups before DR drills.
- Keep customer technical staff present during the test window.
- Use UTC for scheduling.
Test Types
| Test | Purpose | Requirements | Expected impact |
|---|
| Shard failover | Validate shard migration without endpoint rebind. | Replication enabled. | Elevated latency, usually low disruption. |
| Node failure test | Validate behavior when a node fails and endpoints rebind. | Replication enabled. | Temporary disconnects and reconnects. |
| Full-cluster DR drill | Validate recovery from persistence/backup. | Persistence/backups enabled. | Longer outage; plan multi-hour window. |
Verify current plan eligibility and scheduling rules with Redis Support. Historically, these tests require Pro plans and advance scheduling.
Pre-Test Checklist
- Confirm database IDs, names, endpoints, regions, and environment.
- Confirm replication/HA status.
- Confirm persistence and backup status for DR tests.
- Confirm test type and UTC time window.
- Confirm applications use hostnames, not static IPs.
- Review DNS caching behavior, especially Java/JVM DNS cache settings.
- Confirm client retry, reconnect, timeout, and backoff behavior.
- Ensure a representative workload runs during the test.
- Prepare application, Redis Cloud, and infrastructure dashboards.
- Open a Redis Support ticket with requested details.
Support Request Details
Provide:
- Redis Cloud account and subscription IDs.
- Database IDs and endpoints.
- Environment: staging, production, or go-live validation.
- Requested test type: shard failover, node failure, or DR drill.
- Preferred UTC windows.
- Business impact and success criteria.
- Customer contacts who will join the test chat.
- Any co-resident database concerns for node tests.
During the Test
Monitor:
- Application disconnect/reconnect logs.
- p95/p99 latency.
- Error and timeout rates.
- Recovery time against RTO.
- Data consistency and write success.
- Redis Cloud database metrics.
Keep traffic active; idle connections do not prove application resilience.
Client Readiness
| Issue | Fix |
|---|
| App does not reconnect | Add retry and reconnect logic with exponential backoff. |
| DNS cached too long | Lower client/runtime DNS TTL; for Java, review networkaddress.cache.ttl. |
| Failover feels slow | Use sensible connect/socket timeouts, often low seconds. |
| Static IPs used | Switch to Redis Cloud endpoint hostnames. |
| Old client library bugs | Upgrade Lettuce, Jedis, Redisson, or other clients. |
| Retry storm | Add jitter, backoff, and circuit-breaking. |
Post-Test Validation
Confirm:
- Redis Support reports test completion.
- Application traffic resumes normally.
- Latency and errors return to baseline.
- RTO/RPO expectations were met or gaps were documented.
- Logs and metrics are saved for the runbook.
- Follow-up client or architecture changes are tracked.
Escalation Packet
Collect:
- Test type and scheduled UTC window.
- Database IDs, endpoints, and regions.
- HA, persistence, and backup status.
- Application client libraries and timeout/retry settings.
- Metrics before, during, and after failover.
- Application errors and reconnect evidence.
- RTO/RPO results.
- Redis Support ticket number.
