cluster-ops

name	cluster-ops
summary	Couchbase cluster operations — replica configuration, failover, rebalance, server groups, and Multi-Dimensional Scaling (MDS)
description	Couchbase cluster operations — replica configuration, failover, rebalance, server groups, and Multi-Dimensional Scaling (MDS)
allowed-tools	Bash
compatibility	Requires Couchbase Server 7.0+. Server Groups and MDS require Enterprise Edition.
metadata	{"last_verified":"2026-05","min_server_version":"7.0","handoff":[{"condition":"user asks about backup or restore","skill":"backup"},{"condition":"user asks about XDCR replication","skill":"xdcr"},{"condition":"user asks about RBAC or security","skill":"security"},{"condition":"user asks about cluster health, metrics, or alerting","skill":"monitoring"},{"condition":"user asks about Couchbase fundamentals or core concepts","skill":"getting-started"}]}

Couchbase Cluster Operations

Replica configuration, failover, rebalance, and service topology for Couchbase Server clusters.

Replica Configuration

Replicas provide high availability. Each bucket can have 0–3 replicas. Replicas are stored on different nodes from the active vBuckets.

Replica sizing guidelines:

Cluster size	Recommended replicas
1 node	0 (no HA possible)
2–4 nodes	1
5–9 nodes	1 or 2
10+ nodes	1, 2, or 3

# Create a bucket with 1 replica
curl -X POST http://localhost:8091/pools/default/buckets \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "name=myapp&ramQuotaMB=512&replicaNumber=1&bucketType=couchbase"

# Update replicas on an existing bucket
curl -X POST http://localhost:8091/pools/default/buckets/myapp \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "replicaNumber=2"
# Requires a rebalance to take effect

Replica reads: by default, reads go to the active vBucket. Enable replica reads for higher read availability at the cost of potential stale data:

from couchbase.options import GetOptions
from couchbase.replica_reads import ReplicaMode

# Read from any available replica (active or replica)
result = collection.get_any_replica("doc_key")

# Read from all replicas and return first response
result = collection.get_all_replicas("doc_key")

---

Multi-Dimensional Scaling (MDS)

MDS lets you assign different services to different nodes, scaling each service independently.

Node 1: Data Service          (KV storage)
Node 2: Data Service          (KV storage)
Node 3: Index + Query Service (GSI + SQL++)
Node 4: Search Service        (FTS + vector)
Node 5: Analytics Service     (OLAP)

Benefits: Data nodes are not impacted by heavy query or search workloads. Each tier scales independently.

Minimum production topology:

• ≥ 2 Data nodes (for replica support)
• ≥ 2 Index nodes (for index HA)
• ≥ 1 Query node (stateless, easy to scale)

Adding a node with specific services:

# Add a node and assign only the Index + Query services
curl -X POST http://localhost:8091/controller/addNode \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "hostname=new-node:8091&user=Administrator&password=NodePass&services=index,n1ql"

# Then rebalance
curl -X POST http://localhost:8091/controller/rebalance \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "knownNodes=ns_1@node1,ns_1@node2,ns_1@new-node&ejectedNodes="

---

Rebalance

Rebalance redistributes vBuckets and indexes across nodes after topology changes (add/remove node). Always rebalance after adding or removing nodes.

# Trigger a rebalance (after adding/removing nodes)
curl -X POST http://localhost:8091/controller/rebalance \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "knownNodes=ns_1@node1,ns_1@node2,ns_1@node3&ejectedNodes="

# Check rebalance progress
curl http://localhost:8091/pools/default/rebalanceProgress \
  -u Administrator:"$CB_ADMIN_PASSWORD"

# Stop a running rebalance
curl -X POST http://localhost:8091/controller/stopRebalance \
  -u Administrator:"$CB_ADMIN_PASSWORD"

Rebalance impact:

• Data Service: vBuckets migrate between nodes — cluster remains fully available
• Index Service: indexes rebuild or transfer to new nodes (file-based transfer in 7.1+)
• Query Service: waits for in-flight queries/transactions to complete before shutting down on removed nodes
• Eventing: mutation processing redistributes across nodes; no mutations are lost
• Analytics: shadow data redistributes; requires replicas to maintain availability during rebalance

Best practices:

• Rebalance during low-traffic periods for large clusters
• Monitor rebalanceProgress — it shows per-service completion percentage
• Do not add/remove multiple nodes simultaneously unless you understand the impact

---

Failover

Failover removes an unresponsive node from the cluster and promotes replica vBuckets to active.

Automatic Failover

Couchbase can automatically fail over nodes that become unresponsive:

# Enable auto-failover (timeout in seconds, max 3 nodes)
curl -X POST http://localhost:8091/settings/autoFailover \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "enabled=true&timeout=120&maxCount=2"

• Timeout: how long a node must be unresponsive before failover triggers (minimum 5s, default 120s)
• maxCount: maximum nodes that can be auto-failed-over before manual intervention is required (1–3)
• Auto-failover never triggers if it would cause data loss (e.g., no replicas available)

Manual Failover

# Graceful failover (waits for replication to complete — preferred)
curl -X POST http://localhost:8091/controller/startGracefulFailover \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "otpNode=ns_1@node-to-remove"

# Hard failover (immediate — use when node is unresponsive)
curl -X POST http://localhost:8091/controller/failOver \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "otpNode=ns_1@failed-node"

Recovery after failover

After a failed node is repaired and rejoins:

# Full recovery (re-sync all data from scratch)
curl -X POST http://localhost:8091/controller/setRecoveryType \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "otpNode=ns_1@recovered-node&recoveryType=full"

# Delta recovery (only sync changes since failover — faster)
curl -X POST http://localhost:8091/controller/setRecoveryType \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "otpNode=ns_1@recovered-node&recoveryType=delta"

# Then rebalance to complete recovery
curl -X POST http://localhost:8091/controller/rebalance \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "knownNodes=ns_1@node1,ns_1@node2,ns_1@recovered-node&ejectedNodes="

Delta recovery is faster but requires the node's data files to be intact. Use full recovery if the node's disk was corrupted or replaced.

---

Server Groups (Rack / Zone Awareness)

Server Groups map nodes to physical racks, availability zones, or data center rows. Couchbase ensures active and replica vBuckets are placed in different groups — a single rack failure cannot cause data loss.

# Create server groups
curl -X POST http://localhost:8091/pools/default/serverGroups \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "name=rack-1"

curl -X POST http://localhost:8091/pools/default/serverGroups \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "name=rack-2"

# Assign nodes to groups when adding them
curl -X POST "http://localhost:8091/pools/default/serverGroups/<group-uuid>/addNode" \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "hostname=node3:8091&user=Administrator&password=NodePass&services=kv"

Minimum for rack awareness: 2 server groups, each with at least 1 Data node, and at least 1 replica configured on the bucket.

Cloud deployment: map server groups to availability zones (AZ-1, AZ-2, AZ-3). With 3 AZs and 1 replica, a single AZ failure does not cause data loss.

---

Arbiter Nodes (7.6+)

Arbiter nodes run no services but participate in quorum decisions. They enable fast failover with fewer full Data nodes:

• Provide quorum arbitration to prevent split-brain in 2-node clusters
• Enable fast failover (sub-second detection)
• Require no RAM quota for services

# Add an arbiter node (no services)
curl -X POST http://localhost:8091/controller/addNode \
  -u Administrator:"$CB_ADMIN_PASSWORD" \
  -d "hostname=arbiter:8091&user=Administrator&password=ArbiterPass&services="

Useful for 2-node clusters where a third full node is cost-prohibitive.

---

Monitoring Cluster Health

# Node status
curl http://localhost:8091/pools/nodes -u Administrator:"$CB_ADMIN_PASSWORD" \
  | python3 -c "import sys,json; [print(n['hostname'], n['status'], n['clusterMembership']) for n in json.load(sys.stdin)['nodes']]"

# Bucket stats
curl http://localhost:8091/pools/default/buckets/myapp/stats \
  -u Administrator:"$CB_ADMIN_PASSWORD"

# Check for rebalance in progress
curl http://localhost:8091/pools/default/rebalanceProgress -u Administrator:"$CB_ADMIN_PASSWORD"

Via cbsh:

nodes                          # all nodes with status and memory
nodes | where status != "healthy"   # unhealthy nodes
buckets                        # bucket list with RAM usage