disaster-recovery

name	disaster-recovery
summary	Couchbase disaster recovery — RTO/RPO planning, XDCR active-passive and active-active topologies, backup-based recovery, failover runbooks, and recovery testing
description	Couchbase disaster recovery — RTO/RPO planning, XDCR active-passive and active-active topologies, backup-based recovery, failover runbooks, and recovery testing
compatibility	Couchbase Server 7.0+. XDCR requires Enterprise Edition for filtering and conflict resolution.
metadata	{"last_verified":"2026-05","min_server_version":"7.0","handoff":[{"condition":"user needs to configure XDCR replication","skill":"xdcr"},{"condition":"user needs to configure or restore backups","skill":"backup"},{"condition":"user needs to add nodes or configure failover","skill":"cluster-ops"}]}

Couchbase Disaster Recovery

Couchbase provides two complementary DR mechanisms:

Mechanism	RTO	RPO	Use case
XDCR (active-passive)	Minutes	Near-zero (async)	Regional failover, geo-distribution
XDCR (active-active)	Seconds	Near-zero	Multi-region writes
Backup + restore	Hours	Hours (backup interval)	Data corruption, accidental deletion
Intra-cluster failover	Seconds	Zero (synchronous replicas)	Node failure within a cluster

RTO / RPO Definitions

RTO (Recovery Time Objective): maximum acceptable downtime after a failure.

RPO (Recovery Point Objective): maximum acceptable data loss (measured in time).

Intra-cluster Failover (Node Failure)

Couchbase replicates data within a cluster using vBucket replicas. When a node fails:

1. Auto-failover triggers after the configured timeout (default 120 s).
2. Replica vBuckets on surviving nodes are promoted to active.
3. Rebalance redistributes data across remaining nodes.

# Check auto-failover settings
curl -u Administrator:password http://localhost:8091/settings/autoFailover

# Trigger manual failover (if auto-failover is disabled)
couchbase-cli failover \
  --cluster localhost \
  --username Administrator \
  --password password \
  --server-failover cb-node3.example.com:8091

Minimum replicas for zero data loss: 1 replica per bucket (tolerates 1 node failure).

XDCR Active-Passive (Regional DR)

Run a standby cluster in a second region. XDCR replicates all writes from the primary asynchronously.

# Create a remote cluster reference on the primary
couchbase-cli xdcr-setup \
  --cluster localhost \
  --username Administrator \
  --password password \
  --create \
  --xdcr-cluster-name dr-cluster \
  --xdcr-hostname dr-cluster.example.com \
  --xdcr-username Administrator \
  --xdcr-password "$DR_PASSWORD"

# Start replication: primary 'myapp' → DR 'myapp'
couchbase-cli xdcr-replicate \
  --cluster localhost \
  --username Administrator \
  --password password \
  --create \
  --xdcr-cluster-name dr-cluster \
  --xdcr-from-bucket myapp \
  --xdcr-to-bucket myapp

Failover runbook (active-passive):

1. Verify DR cluster has caught up: check xdcr_changes_left metric → should be near 0.
2. Update DNS / load balancer to point to DR cluster.
3. Redirect application connection strings to DR cluster.
4. Disable XDCR replication from primary (it may be unreachable).
5. After primary recovers: re-establish XDCR in reverse to sync changes back, then cut over.

XDCR Active-Active (Multi-Region Writes)

Both clusters accept writes. XDCR replicates bidirectionally. Conflict resolution uses timestamp-based last-write-wins by default; custom conflict resolution requires Enterprise Edition.

See the xdcr skill for full configuration details, filtering, and conflict resolution options.

Backup-Based Recovery

Use cbbackupmgr for point-in-time recovery — restoring to a specific snapshot after data corruption or accidental deletion. This does not replace XDCR for regional failover (RTO is hours, not minutes).

See the backup skill for backup configuration, restore commands, and scheduling.

Recovery decision tree:

• Node failure → intra-cluster failover (auto or manual)
• Region failure → XDCR active-passive failover
• Data corruption / accidental delete → cbbackupmgr restore to point-in-time snapshot
• Full cluster loss with no DR → restore from backup (highest RTO)

Recovery Testing

DR plans that are never tested fail when needed. Recommended cadence:

Test	Frequency	What to verify
Auto-failover	Quarterly	Node removed → auto-failover fires → rebalance completes
XDCR failover	Semi-annually	Cut traffic to DR cluster → application works → cut back
Backup restore	Monthly	Restore latest snapshot to staging → verify document counts

# Check XDCR replication lag (changes not yet replicated)
curl -u Administrator:password \
  http://localhost:8091/pools/default/buckets/myapp/stats \
  | grep xdcr_changes_left

# Verify backup integrity before a DR test
cbbackupmgr examine \
  --archive /backup-archive \
  --repo my-cluster-repo

Key Metrics to Monitor

Metric	Alert threshold	Meaning
xdcr_changes_left	> 10 000	XDCR falling behind — check network / target cluster
ep_num_active_vbuckets	< expected	vBuckets not active — failover may be needed
cbbackupmgr list age	> RPO	Last backup older than RPO target