| name | ha-topology-and-failover |
| description | Design a production database's high-availability topology from its RPO/RTO targets — replica type (sync/async/quorum), failure-domain spread (multi-AZ/region), and the failover mechanism with a split-brain guard. Reach for this when a DB needs to survive an AZ/region outage, when failover has never been designed, or when 'we have a replica' is mistaken for HA. Pairs with backup-and-restore-verification. |
Skill: HA topology & failover
Design availability from the targets down, and design the failover, not just
the replica.
Step 0 — One opinion up front
A replica is not HA; a tested failover is. A standby you can't promote safely
under load is decoration. The deliverable is the promotion path + split-brain guard,
not the replica count.
Step 1 — State the targets
Write down RTO (how long can we be down?), RPO (how much data can we lose?),
and the failure domains you must survive (node → AZ → region). No targets, no
design.
Step 2 — Trace the topology tree
Traverse ../../knowledge/dbre-decision-trees.md §1
to a leaf. Record the path and the runner-up. Key forks: async vs sync (RPO vs write
latency), single-AZ vs multi-AZ vs multi-region (which outage you survive),
automatic vs manual failover (RTO vs complexity).
Step 3 — Design the failover mechanism
Specify explicitly:
- Promotion: what promotes a replica (orchestrator / managed control plane).
- Split-brain guard: fencing / quorum / STONITH so two primaries can't both accept writes.
- Trigger: who or what initiates, and the health signal it uses.
- App reconnection: how clients discover the new primary (DNS, proxy, service discovery).
Step 4 — Place across failure domains
Confirm primary, replicas, and backups live in different domains. A replica in the
primary's AZ doesn't survive an AZ outage.
Step 5 — Prove it (game-day)
A topology is a hypothesis until failover is rehearsed. Hand the drill to
database-operations-engineer (the ha-topology-and-failover design pairs with an
operational failover drill).
Step 6 — Hand off
- The backup/PITR that complements HA →
backup-and-restore-verification.
- Provisioning the instances/replicas →
terraform-iac / cloud plugins.
- The schema the topology carries →
database-engineering.
Output
An HA topology with the tree path + runner-up, RPO/RTO it meets, the failover
mechanism (promotion + split-brain guard + trigger + app reconnection), the
failure-domain placement, and a game-day plan to prove it.
