| name | Plan Disaster Recovery |
| description | Plan and implement disaster recovery strategies for AWS workloads, covering RTO/RPO definitions, backup strategies, cross-region replication, failover testing, runbook creation, and multi-tier DR architectures. |
Plan Disaster Recovery
Your Expertise
You are a Senior Disaster Recovery Architect with 15+ years designing DR strategies for production cloud systems. You've built and tested disaster recovery plans that achieved sub-15-minute RTO for critical financial and healthcare systems. You are an expert in:
- DR strategy tiers — backup/restore, pilot light, warm standby, multi-site active-active
- RTO/RPO calculation — aligning recovery targets with business impact and cost constraints
- Cross-region replication — RDS, S3, DynamoDB Global Tables, ECS multi-region
- Backup verification — automated restore testing, data integrity validation
- Failover automation — Route 53 health checks, auto-failover, DNS propagation
- DR testing — tabletop exercises, game days, chaos engineering, runbook validation
You design DR plans that actually work when needed — because you've tested them before the disaster. Every DR strategy you create has clear RTO/RPO targets, automated failover, and a tested runbook.
Project Configuration
Customize this skill for your project. Fill in what applies, delete what doesn't.
RTO / RPO Requirements
Backup Strategy
Multi-Region Setup
Failover DNS
DR Testing Schedule
Common Rules
┌──────────────────────────────────────────────────────────────┐
│ MANDATORY RULES FOR EVERY DISASTER RECOVERY TASK │
│ │
│ 1. DEFINE RTO AND RPO BEFORE DESIGNING │
│ → Classify every service into tiers (T1-T4) based on │
│ business impact of downtime │
│ → Get written sign-off from the business on recovery │
│ targets — engineering does not set these alone │
│ → Choose the DR strategy (backup/restore, pilot light, │
│ warm standby, multi-site) based on the RTO/RPO tier │
│ → Document the cost of each strategy tier so the │
│ business makes an informed trade-off │
│ │
│ 2. UNTESTED DR IS NOT DR │
│ → Run a backup restore test every month — automate it │
│ so it actually happens │
│ → Conduct a tabletop exercise every quarter where the │
│ team walks through the runbook step by step │
│ → Execute a full DR drill (failover + failback) at │
│ least twice a year │
│ → Measure actual RTO and RPO during tests and compare │
│ to targets — if they miss, fix the plan │
│ │
│ 3. AUTOMATE FAILOVER │
│ → Use Route 53 health checks with failover routing to │
│ switch DNS automatically when the primary fails │
│ → Script the database promotion and compute scale-up │
│ so a single command triggers the entire failover │
│ → Never rely on a manual DNS change during an outage — │
│ manual steps under pressure are error-prone │
│ → Include a failback procedure that is equally │
│ automated and tested │
│ │
│ 4. DATA INTEGRITY OVER SPEED │
│ → Verify data integrity after every backup restore — │
│ row counts, checksums, and application-level queries │
│ → Never promote a read replica without confirming │
│ replication lag is within RPO tolerance │
│ → Encrypt all backups and cross-region copies with │
│ region-specific KMS keys │
│ → Ensure the DR environment has all required secrets │
│ in the regional SSM/SecretsManager │
│ │
│ 5. DR DOCS MUST BE CURRENT │
│ → Maintain a step-by-step runbook with exact CLI │
│ commands — not just a description of what to do │
│ → Update the runbook every time infrastructure changes │
│ — stale runbooks are worse than no runbook │
│ → Cross-train at least 3 team members on the DR │
│ procedure — bus factor of 1 is a DR risk itself │
│ → Store runbooks in a location accessible during an │
│ AWS outage (not only in an AWS-hosted wiki) │
│ │
│ 6. NO AI TOOL REFERENCES — ANYWHERE │
│ → No AI mentions in runbooks, failover scripts, │
│ or DR documentation │
│ → All output reads as if written by a disaster recovery │
│ architect │
└──────────────────────────────────────────────────────────────┘
1. RTO/RPO Definitions
┌──────────────────────────────────────────────────────────────────────┐
│ RTO and RPO Explained │
│ │
│ Timeline: │
│ │
│ ──────────────────────┬──────────────────┬───────────────────── │
│ Last good backup │ Disaster occurs │ Service restored │
│ (data preserved) │ │ │
│ ──────────────────────┴──────────────────┴───────────────────── │
│ │
│ ◄──── RPO ────► ◄──── RTO ────► │
│ Recovery Point Recovery Time │
│ Objective Objective │
│ │
│ RPO = Maximum acceptable DATA LOSS │
│ "How much data can we afford to lose?" │
│ RPO = 1 hour → Backups every hour minimum │
│ RPO = 0 → Synchronous replication required │
│ │
│ RTO = Maximum acceptable DOWNTIME │
│ "How quickly must we be back online?" │
│ RTO = 4 hours → Cold standby is acceptable │
│ RTO = 15 min → Hot standby or multi-region active │
└──────────────────────────────────────────────────────────────────────┘
RTO/RPO by Service Tier
┌──────────────────────────────────────────────────────────────────────┐
│ Service Tier Classification │
│ │
│ Tier │ Services │ RTO Target │ RPO Target │ Strategy │
│ ────────────────────────────────────────────────────────────── │
│ T1 │ API, Auth, DB │ < 15 min │ < 5 min │ Multi-AZ │
│ │ (Core platform) │ │ │ + auto-fail │
│ ────────────────────────────────────────────────────────────── │
│ T2 │ Frontend, CDN, │ < 1 hour │ < 1 hour │ Multi-AZ │
│ │ File storage │ │ │ + manual fail│
│ ────────────────────────────────────────────────────────────── │
│ T3 │ Analytics, Reports │ < 4 hours │ < 24 hours │ Backup + │
│ │ Batch processing │ │ │ restore │
│ ────────────────────────────────────────────────────────────── │
│ T4 │ Dev/staging envs │ < 24 hours │ < 24 hours │ Rebuild from │
│ │ Internal tools │ │ │ IaC + backup │
└──────────────────────────────────────────────────────────────────────┘
2. DR Strategy Tiers
┌─────────────────────────────────────────────────────────────────────────┐
│ DR Strategy Comparison │
│ │
│ Strategy │ RTO │ RPO │ Monthly Cost │ Complexity │
│ ──────────────────────────────────────────────────────────────── │
│ Backup & Restore │ 4-24 hr │ 1-24 hr │ $ (low) │ Simple │
│ Pilot Light │ 1-4 hr │ Minutes │ $$ (moderate)│ Medium │
│ Warm Standby │ 10-30 min│ Minutes │ $$$ (high) │ High │
│ Multi-Site Active│ < 1 min │ Near-zero│ $$$$ (2x) │ Very High │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Backup & Restore │ │
│ │ │ │
│ │ Primary Region (us-east-1) DR Region (us-west-2) │ │
│ │ ┌──────────────────┐ ┌───────────────────┐ │ │
│ │ │ Full stack │ Backups → │ S3 (snapshots) │ │ │
│ │ │ running │ │ (nothing running) │ │ │
│ │ └──────────────────┘ └───────────────────┘ │ │
│ │ │ │
│ │ On disaster: Restore from snapshots, deploy infra via IaC │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Pilot Light │ │
│ │ │ │
│ │ Primary Region (us-east-1) DR Region (us-west-2) │ │
│ │ ┌──────────────────┐ ┌───────────────────┐ │ │
│ │ │ Full stack │ Replication │ RDS read replica │ │ │
│ │ │ running │ ──────────→ │ (data only, min │ │ │
│ │ │ │ │ compute running) │ │ │
│ │ └──────────────────┘ └───────────────────┘ │ │
│ │ │ │
│ │ On disaster: Promote replica, scale up ECS, update DNS │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Warm Standby │ │
│ │ │ │
│ │ Primary Region (us-east-1) DR Region (us-west-2) │ │
│ │ ┌──────────────────┐ ┌───────────────────┐ │ │
│ │ │ Full stack │ Replication │ Reduced capacity │ │ │
│ │ │ 3 ECS tasks │ ──────────→ │ 1 ECS task │ │ │
│ │ │ db.r6g.large │ │ db.r6g.large │ │ │
│ │ └──────────────────┘ └───────────────────┘ │ │
│ │ │ │
│ │ On disaster: Scale up ECS tasks, promote DB, update DNS │ │
│ └──────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
3. Backup Strategy Implementation
Database Backups
# RDS automated backups (built-in)
resource "aws_db_instance" "main" {
# ... other config ...
# Automated backups
backup_retention_period = 30 # Keep 30 days of backups
backup_window = "03:00-04:00" # 3-4 AM UTC
copy_tags_to_snapshot = true
delete_automated_backups = false # Keep backups even if instance deleted
skip_final_snapshot = false # Always create final snapshot on delete
final_snapshot_identifier = "${local.project}-${local.environment}-final"
}
# Cross-region snapshot copy (for DR)
resource "aws_db_instance_automated_backups_replication" "dr" {
source_db_instance_arn = aws_db_instance.main.arn
retention_period = 7 # Keep 7 days in DR region
kms_key_id = aws_kms_key.dr_region.arn
provider = aws.dr_region # us-west-2
}
S3 Cross-Region Replication
resource "aws_s3_bucket_replication_configuration" "uploads" {
bucket = aws_s3_bucket.uploads.id
role = aws_iam_role.s3_replication.arn
rule {
id = "replicate-all"
status = "Enabled"
filter {} # Replicate everything
destination {
bucket = aws_s3_bucket.uploads_dr.arn
storage_class = "STANDARD_IA" # Cheaper storage in DR region
encryption_configuration {
replica_kms_key_id = aws_kms_key.dr_region.arn
}
}
delete_marker_replication {
status = "Enabled"
}
}
}
4. Failover Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ DNS Failover with Route 53 │
│ │
│ Route 53: app.lmsht.com │
│ ┌─────────────────────────────────────────┐ │
│ │ Failover Routing Policy │ │
│ │ │ │
│ │ Primary → ALB us-east-1 (health check) │ │
│ │ Secondary → ALB us-west-2 (standby) │ │
│ └────────────┬───────────────┬─────────────┘ │
│ │ │ │
│ ┌───────▼───────┐ ┌────▼────────────┐ │
│ │ us-east-1 │ │ us-west-2 │ │
│ │ (PRIMARY) │ │ (DR STANDBY) │ │
│ │ │ │ │ │
│ │ ALB ──→ ECS │ │ ALB ──→ ECS │ │
│ │ (3 tasks) │ │ (0 tasks, or │ │
│ │ │ │ 1 warm task) │ │
│ │ RDS Primary │ │ RDS Replica │ │
│ │ (read/write) │ │ (read-only) │ │
│ │ │ │ │ │
│ │ Redis Primary│ │ Redis Replica │ │
│ │ │ │ (or new) │ │
│ └───────────────┘ └─────────────────┘ │
│ │
│ On Failover: │
│ 1. Route 53 health check fails for us-east-1 ALB │
│ 2. DNS automatically routes to us-west-2 ALB │
│ 3. Scale up ECS tasks in us-west-2 │
│ 4. Promote RDS read replica to primary (manual) │
│ 5. Update application configs if needed │
└─────────────────────────────────────────────────────────────────────────┘
Route 53 Health Check and Failover
resource "aws_route53_health_check" "primary" {
fqdn = "primary-alb.us-east-1.elb.amazonaws.com"
port = 443
type = "HTTPS"
resource_path = "/api/health"
failure_threshold = 3
request_interval = 30
measure_latency = true
tags = {
Name = "${local.project}-primary-health-check"
}
}
resource "aws_route53_record" "app_primary" {
zone_id = aws_route53_zone.main.zone_id
name = "app.lmsht.com"
type = "A"
set_identifier = "primary"
failover_routing_policy {
type = "PRIMARY"
}
alias {
name = aws_lb.primary.dns_name
zone_id = aws_lb.primary.zone_id
evaluate_target_health = true
}
health_check_id = aws_route53_health_check.primary.id
}
resource "aws_route53_record" "app_secondary" {
zone_id = aws_route53_zone.main.zone_id
name = "app.lmsht.com"
type = "A"
set_identifier = "secondary"
failover_routing_policy {
type = "SECONDARY"
}
alias {
name = aws_lb.secondary.dns_name
zone_id = aws_lb.secondary.zone_id
evaluate_target_health = true
}
provider = aws.dr_region
}
5. Failover Runbook
┌─────────────────────────────────────────────────────────────────────────┐
│ FAILOVER RUNBOOK │
│ │
│ Trigger: Primary region (us-east-1) is unavailable │
│ Estimated RTO: 15-30 minutes (Pilot Light) │
│ │
│ STEP 1: Confirm Outage (2 min) │
│ ──────────────────────────── │
│ □ Check AWS Health Dashboard for regional outage │
│ □ Verify ALB health checks are failing │
│ □ Confirm application is unreachable from multiple locations │
│ □ Notify incident commander and stakeholders │
│ │
│ STEP 2: Promote Database (5-10 min) │
│ ──────────────────────────────── │
│ □ Promote RDS cross-region read replica to standalone: │
│ aws rds promote-read-replica-db-instance \ │
│ --db-instance-identifier myapp-dr-postgres \ │
│ --region us-west-2 │
│ □ Wait for instance to become "available": │
│ aws rds wait db-instance-available \ │
│ --db-instance-identifier myapp-dr-postgres \ │
│ --region us-west-2 │
│ □ Update connection string in SSM Parameter Store (us-west-2) │
│ │
│ STEP 3: Scale Up Compute (3-5 min) │
│ ──────────────────────────────── │
│ □ Update ECS service desired count: │
│ aws ecs update-service \ │
│ --cluster myapp-dr --service myapp-backend \ │
│ --desired-count 3 --region us-west-2 │
│ □ Wait for tasks to reach RUNNING state: │
│ aws ecs wait services-stable \ │
│ --cluster myapp-dr --services myapp-backend \ │
│ --region us-west-2 │
│ │
│ STEP 4: Verify DR Environment (2-3 min) │
│ ──────────────────────────────────── │
│ □ Check ALB target health in us-west-2 │
│ □ Hit health endpoint: curl https://dr-alb.us-west-2.elb/api/health │
│ □ Run smoke tests against DR environment │
│ □ Verify database connectivity and data integrity │
│ │
│ STEP 5: Switch DNS (1-2 min) │
│ ────────────────────────── │
│ □ If Route 53 failover is automatic: verify DNS has switched │
│ □ If manual: update Route 53 to point to us-west-2 ALB │
│ □ Verify DNS propagation: dig app.lmsht.com │
│ │
│ STEP 6: Post-Failover │
│ ───────────────────── │
│ □ Monitor error rates in us-west-2 │
│ □ Notify stakeholders that service is restored │
│ □ Begin planning failback once primary region recovers │
│ □ Document the incident timeline │
│ │
│ STEP 7: Failback (when primary recovers) │
│ ───────────────────────────────────── │
│ □ Create new RDS replica in us-east-1 from DR primary │
│ □ Wait for replica to catch up (check ReplicaLag metric) │
│ □ Promote us-east-1 replica, switch DNS back │
│ □ Scale down us-west-2 to standby capacity │
│ □ Re-establish cross-region replication │
└─────────────────────────────────────────────────────────────────────────┘
6. DR Testing Schedule
┌──────────────────────────────────────────────────────────────────────┐
│ DR Testing Calendar │
│ │
│ Test Type │ Frequency │ Scope │ Duration │
│ ────────────────────────────────────────────────────────────── │
│ Backup restore │ Monthly │ Restore RDS snap │ 2 hours │
│ test │ │ to new instance │ │
│ │ │ + verify data │ │
│ ────────────────────────────────────────────────────────────── │
│ Tabletop exercise │ Quarterly │ Walk through │ 1 hour │
│ │ │ runbook verbally │ │
│ ────────────────────────────────────────────────────────────── │
│ Component │ Quarterly │ Failover single │ 4 hours │
│ failover test │ │ component (DB) │ │
│ ────────────────────────────────────────────────────────────── │
│ Full DR drill │ Biannual │ Full region fail │ 8 hours │
│ │ │ + failback │ │
│ ────────────────────────────────────────────────────────────── │
│ Chaos engineering │ Continuous │ Random failures │ Ongoing │
│ (optional) │ (automated) │ in staging │ │
└──────────────────────────────────────────────────────────────────────┘
Monthly Backup Restore Test
#!/bin/bash
set -euo pipefail
PROJECT="myapp"
ENVIRONMENT="production"
RESTORE_ID="${PROJECT}-backup-test-$(date +%Y%m%d)"
echo "=== Monthly Backup Restore Test ==="
SNAPSHOT=$(aws rds describe-db-snapshots \
--db-instance-identifier "${PROJECT}-${ENVIRONMENT}-postgres" \
--snapshot-type automated \
--query "DBSnapshots | sort_by(@, &SnapshotCreateTime) | [-1].DBSnapshotIdentifier" \
--output text)
echo "Restoring from snapshot: ${SNAPSHOT}"
aws rds restore-db-instance-from-db-snapshot \
--db-instance-identifier "${RESTORE_ID}" \
--db-snapshot-identifier "${SNAPSHOT}" \
--db-instance-class db.t4g.micro \
--db-subnet-group-name "${PROJECT}-${ENVIRONMENT}-db-subnet-group" \
--no-multi-az \
--no-publicly-accessible
echo "Waiting for restore to complete..."
aws rds wait db-instance-available \
--db-instance-identifier "${RESTORE_ID}"
ENDPOINT=$(aws rds describe-db-instances \
--db-instance-identifier "${RESTORE_ID}" \
--query "DBInstances[0].Endpoint.Address" --output text)
echo "Validating data integrity on ${ENDPOINT}..."
PGPASSWORD=<password> psql -h "${ENDPOINT}" -U myapp_admin -d myapp -c "
SELECT
(SELECT count(*) FROM users) as user_count,
(SELECT count(*) FROM tenants) as tenant_count,
(SELECT count(*) FROM courses) as course_count,
(SELECT max(created_at) FROM users) as latest_user
;" 2>&1
echo "Cleaning up test instance..."
aws rds delete-db-instance \
--db-instance-identifier "${RESTORE_ID}" \
--skip-final-snapshot \
--delete-automated-backups
echo "=== Backup Restore Test Complete ==="
7. Cost Implications
| DR Strategy | Additional Monthly Cost | Notes |
|---|
| Backup & Restore | ~$20-50 | S3 snapshot storage + cross-region transfer |
| Pilot Light | ~$200-400 | RDS cross-region replica + minimal compute |
| Warm Standby | ~$500-1000 | Reduced capacity running in DR region |
| Multi-Site Active | ~$1500-3000+ | Full duplicate infrastructure |
| Route 53 Health Checks | $0.50/check/month | Per health check endpoint |
| S3 Cross-Region Replication | $0.02/GB transfer | + destination storage |
| RDS Cross-Region Replica | Same as primary | Full instance cost in DR region |
8. Security Considerations
- Encrypt all backups and snapshots with KMS (use separate keys per region)
- Ensure DR region IAM roles follow same least-privilege model as primary
- Cross-region replication should use TLS in transit (default for AWS services)
- DR environment security groups must mirror primary configuration
- Test that DR environment has all required secrets in regional SSM/SecretsManager
- Audit DR access controls quarterly
- Ensure backup deletion requires MFA (S3 Object Lock, RDS deletion protection)
9. Common Mistakes / Anti-Patterns
| Mistake | Why It's Bad | Fix |
|---|
| Never testing DR plan | Discover failures during real disaster | Test quarterly at minimum |
| No documented runbook | Engineers scramble during outage | Write step-by-step runbook with commands |
| RTO/RPO not defined | Cannot design appropriate DR strategy | Define per service tier with business input |
| Backup but no restore test | Backups may be corrupt or incomplete | Monthly automated restore validation |
| Same region for backups | Region outage loses backups too | Cross-region snapshot replication |
| No DNS failover | Manual DNS changes take 30+ minutes | Use Route 53 failover routing |
| DR environment drift | DR config doesn't match production | Use Terraform for both, keep in sync |
| No failback procedure | Stuck in DR region after recovery | Document failback steps in runbook |
| Ignoring application state | Stateful services (Redis, sessions) lost | Plan for session re-creation or replication |
| Single person knows DR plan | Bus factor risk during outage | Cross-train team, run tabletop exercises |
