mongodb-replication-and-ha-readiness

name

mongodb-replication-and-ha-readiness

description

Use when designing or hardening MongoDB replica-set topology and high-availability posture after the data model exists and reliability and data-architecture have set the availability and consistency targets. Produces replica-set member topology and election behavior, read-preference routing, write-concern selection that the topology can actually honor, arbiter posture, oplog-window sizing for failover, change-stream availability, and multi-region replica placement. Do not use for document modeling or shard-key choice, query tuning, backup/PITR, security/auth, or host/cluster provisioning; use the other Family B archetype skills (provisioning is infrastructure).

MongoDB Replication and HA Readiness

When to use

Invoke when standing up a replica set for production, restructuring members/elections, validating that the declared write/read concerns are actually survivable, sizing the oplog for failover, or placing replicas across regions.

Do not use for: document modeling, validators, or shard-key choice (use mongodb-data-model-and-migration); query/index tuning (use mongodb-indexing-and-query-optimization); backup, oplog-based PITR, restore drills (use mongodb-backup-and-operational-readiness); auth/RBAC/TLS/audit and internal-auth credentials (use mongodb-security-and-data-access-hardening); host, VM, or cluster provisioning (infrastructure layer — this skill designs the MongoDB topology, not the machines).

Inputs

Required:

• An existing data model from mongodb-data-model-and-migration (the per-operation read/write concern declarations this topology must be able to honor).
• Approved architecture/reliability availability targets and RPO/RTO per data tier, or explicit confirmation they are intentionally deferred.

Optional:

• Approved data-architecture.md consistency posture and replica/region intent.
• The workload tier from architecture-schema (drives member count, region spread, arbiter prohibition).
• Read/write mix and latency sensitivity per operation class.
• Region/AZ topology available and inter-region latency.
• Change-stream consumers (resumability requirements) and their lag tolerance.

Operating rules

• Never design a topology the declared concerns cannot survive. If the model declares w: "majority", the member topology must keep a writable majority through the planned failure (one node, one AZ, one region per the tier). A topology that cannot honor the declared concern is rejected.
• Consume architecture/reliability and data-architecture.md; do not invent decisions. Availability targets, RPO/RTO, and consistency posture are upstream. If a needed decision is missing, pause and raise an ADR candidate rather than guessing.
• Design the topology; do not redesign the model. Per-operation read/write concern is declared by mongodb-data-model-and-migration. This skill ensures the replica set can honor it; a concern that is wrong for the durability requirement is a finding handed back, not changed here.
• Odd voting-member count; elections must always resolve. An even voting count or a partition-prone layout that can deadlock an election is rejected. Member priorities reflect intended primary placement.
• Arbiters are a last resort, not a default. An arbiter contributes a vote but no data — with w: "majority" it can force acknowledged-but-not-durable writes on a member loss. Arbiters are prohibited for tier-0 and used elsewhere only with an explicit ADR.
• Read preference is explicit and consistency-aware. primary for read-after-write; secondaryPreferred/nearest only where staleness is acceptable and stated. Every secondary-read path names its staleness tolerance; maxStalenessSeconds is set where used.
• Write concern is sized to durability, not latency convenience. State-mutating writes default to w: "majority" (and j: true where the loss of the last write is unacceptable). A relaxed concern names the data it applies to and the accepted loss window.
• The oplog window must exceed the worst recovery interval. Size the oplog so a lagging or recovering secondary (and the failover/maintenance window) fits inside it; an oplog that rolls over before a secondary catches up forces a full resync. State the measured window vs the requirement.
• Multi-region placement matches the tier and the RTO. Tier-0 cross-region posture (and whether a region can be lost without data loss given the write concern) is explicit, including the latency cost of w: "majority" across regions.
• Change-stream availability is designed, not assumed. If consumers depend on change streams, resumability (resume tokens), the oplog window backing them, and behavior across failover are specified.
• This skill owns replica-set topology + HA posture. Modeling, query tuning, backup/PITR, security/auth, and host provisioning are named handoffs; internal-auth (keyfile/x.509) is required and named to .

Output contract

The replication and HA design MUST conform to:

• observability-standards — replication lag, election/stepdown events, oplog window, and member health are exposed signals.
• deployment-standards — the topology is reproducible from configuration, not click-ops; member changes are rolling.
• architecture-schema — tier classification drives member count, region spread, and arbiter prohibition.
• naming-conventions — replica-set and member naming.
• security-standards — internal auth (keyfile/x.509) and TLS between members are required; the mechanism is named to mongodb-security-and-data-access-hardening.

Upstream contract: architecture/reliability is the source of truth for availability targets and RPO/RTO; data-architecture.md is the source of truth for consistency posture. Per-operation read/write concern is declared by mongodb-data-model-and-migration. If a needed decision is missing, pause and raise an ADR candidate.

Progressive references

• Read references/mongodb-replication-playbook.md when designing any owned area or checking the anti-pattern list.
• Read references/mongodb-replication-quality-rubric.md before declaring the topology complete.
• Use assets/mongodb-replication.template.md as the topology / read-preference / write-concern / oplog pattern reference.

Process

1. Gather context: load architecture/reliability (availability, RPO/RTO) and data-architecture.md (consistency, region intent). Pull the per-operation read/write concern declarations from mongodb-data-model-and-migration. Resolve the tier from architecture-schema. If a needed decision is missing, raise an ADR candidate before proceeding.
2. Design member topology: odd voting count, data-bearing members across the AZ/region spread the tier requires, member priorities reflecting intended primary placement, hidden/delayed members if the reliability posture calls for them.
3. Validate concern survivability: confirm the declared w/readConcern is still honored through the planned failure (one node / one AZ / one region per tier). Where it is not, raise a finding to mongodb-data-model-and-migration or an ADR candidate.
4. Set arbiter posture: prohibit for tier-0; elsewhere only with an explicit ADR documenting the durability trade-off under w: "majority".
5. Define read-preference routing per operation class: primary for read-after-write, secondaryPreferred/nearest only with a stated staleness tolerance and maxStalenessSeconds.
6. Confirm write-concern posture matches durability: w: "majority" (+ j: true where required) for state mutations; any relaxed concern names the data and the accepted loss window.
7. Size the oplog: compute the worst recovery/lag/maintenance interval and set the oplog window to exceed it; state measured-vs-required.
8. Design multi-region placement (if in scope): member roles per region, the latency cost of cross-region majority, and whether a region loss is survivable without data loss given the write concern.
9. Specify change-stream availability: resume-token handling, the oplog window backing streams, and failover behavior for consumers.
10. Rehearse failover: step down or kill the primary; confirm a timely election, the declared concerns still honored, no acknowledged-write loss; capture timings vs the RTO. Document any check that cannot run.
11. Produce replication-ha.md (member/priority/region map, concern-survivability matrix, read-preference table, oplog sizing, change-stream posture, rehearsed-failover results) plus the named handoff list. Validate against observability-, deployment-, security-standards, architecture-schema, and naming-conventions. Revise until all pass or the gap is documented.

Outputs

Required:

• Replica-set member topology (counts, priorities, AZ/region placement, hidden/delayed if used).
• Concern-survivability matrix: declared w/readConcern vs the planned failure.
• Read-preference routing table with staleness tolerances and maxStalenessSeconds.
• Write-concern posture per operation class with accepted-loss windows for any relaxed concern.
• Oplog sizing (measured worst recovery interval vs configured window).
• Multi-region placement and change-stream availability design (where in scope).
• Rehearsed-failover results vs RTO.
• replication-ha.md and the named handoff list.

Output rules:

• Reproducible topology configuration — not prose-only, not click-ops.
• Odd voting count; no tier-0 arbiter; no topology that cannot honor the declared concern.
• Every secondary-read path states its staleness tolerance.
• Modeling, query tuning, backup/PITR, security/auth, and host provisioning are named handoffs.

Quality checks

• Availability targets and RPO/RTO are sourced from architecture/reliability; consistency posture from data-architecture.md (or an ADR candidate is raised).
• The declared per-operation read/write concerns come from mongodb-data-model-and-migration; this skill does not redefine them.
• Voting-member count is odd and elections always resolve; member priorities reflect intended primary placement.
• No arbiter for tier-0; any arbiter elsewhere has an ADR documenting the w:"majority" durability trade-off.
• The declared write/read concern is honored through the planned failure (node/AZ/region per tier); a gap is a finding or ADR candidate.
• Every secondary-read path names a staleness tolerance and sets maxStalenessSeconds.
• State-mutating writes use w: "majority" (+ j: true where required); relaxed concerns name the data and accepted loss window.
• The oplog window exceeds the worst recovery/lag/maintenance interval (measured vs required stated).
• Multi-region placement (if in scope) states the cross-region majority latency cost and region-loss survivability.
• Change-stream resumability and failover behavior are specified where consumers exist.
• Failover was rehearsed (primary stepdown/kill) with election timing, concern still honored, and no acknowledged-write loss, or the gap is documented.

References

• Upstream: architecture/reliability, architecture/data-architecture.
• Builds on: mongodb-data-model-and-migration (declares the per-operation read/write concerns this topology must honor).
• Related Family B archetype skills: mongodb-indexing-and-query-optimization, mongodb-backup-and-operational-readiness (oplog-based PITR/restore), mongodb-security-and-data-access-hardening (internal auth + TLS between members).
• Downstream: backend implementation skills own read-preference wiring in ODM/repository code.
• Standards: observability-standards, deployment-standards, architecture-schema, naming-conventions, security-standards.