recover-from-a-cross-region-data-consistency-failure

name	recover-from-a-cross-region-data-consistency-failure
category	code
description	Recover from cross-region data inconsistency through write fencing, immutable evidence, causal and business-state reconciliation, customer repair, safe authority restoration, and prevention. Use when partitions, replication, failover, clocks, or conflicting writes corrupt shared state.

recover-from-a-cross-region-data-consistency-failure

Restore one authoritative history before resuming unrestricted writes.

When to use

• Use for split brain, stale leader, failed failover, replication gap, clock anomaly, conflict-resolution defect, or partial regional restore.
• Activate incident, data, application, finance, identity, security, legal, support, and regional authority.

Preconditions

• Establish one incident authority, trusted time sources, write-stop or per-domain fencing, source custody, customer protection, and reversible repair.
• Define regions, stores, schemas, replication, clocks, leaders, conflicts, consumers, business invariants, and recovery objectives.

Procedure

Complete containment and evidence preservation, causal and business reconciliation, authoritative repair, and safe restart.

1. Open a cross-region consistency incident timeline for leaders, epochs, writes, replication, clocks, failures, decisions, and customer impact.
2. Fence writes by domain, key range, tenant, or operation; prevent two regions from acting on the same external side effect.
3. Preserve immutable logs, change streams, transaction records, snapshots, offsets, conflict decisions, application versions, clocks, routing, and external acknowledgements.
4. Establish canonical operation identity and reconstruct causal order from epochs, versions, transaction IDs, source sequence, observed time, and durable outcomes rather than wall clock alone.
5. Classify every affected entity as consistent, missing, duplicate, divergent, stale, orphaned, conflicted, or externally applied.
6. Validate business invariants across money, inventory, identity, permissions, orders, messages, and downstream products.
7. Select authoritative state per domain with legal and business owners; preserve contrary versions and reasons.
8. Build repair in an isolated namespace from immutable evidence, replaying idempotently and reconciling external side effects before publication.
9. Test repair against every region, downstream consumer, customer statement, audit, and historical query.
10. Assign one repair generation and authority epoch across all affected domains; every write, job, event, cache entry, and downstream publication carries it and every enforcement point rejects stale epochs.
11. Establish per-domain high-watermarks, fence linked operations, require all domains prepared, then activate through one independently controlled decision.
12. Journal every post-cutover write and external effect; rollback fences writes and creates or reconciles a new generation rather than blindly reactivating an old one.
13. Publish through canary reads and writes, monitoring, and the generation-fenced rollback protocol.
14. Correct customer-visible records and external systems, notify as required, and verify remediation.
15. Rebuild fencing, leader election, replication, conflict, clock, failover, invariant, and rehearsal controls before removing restrictions.

Failure plan

• If causal order or external outcome is ambiguous, stop that entity and route to owned adjudication.
• If repair would overwrite unique evidence, write a new generation and preserve all prior state.
• If a region cannot reject stale epochs, keep it read-only or isolated.

Worked example

A network partition and automated failover leave two regions accepting orders, payments, inventory reservations, account changes, and refunds under different clocks; replication later overwrites some newer records while external payment and shipping systems retain both effects. Recovery fences writes, reconstructs operation and causal history, reconciles external outcomes, repairs a new state generation, corrects customers, and proves safe authority.

Done

• A cross-region consistency incident timeline verifies epochs, leaders, writes, replication, clocks, evidence, decisions, external outcomes, and customer impact
• A causal, entity, business-invariant, and downstream reconciliation proves authoritative disposition for missing, duplicate, divergent, stale, conflicted, and externally applied state
• A repair and restart report demonstrates isolated reconstruction, idempotent replay, one cross-domain generation epoch, post-cutover write and external-effect journaling, regional stale-epoch rejection, customer remediation, generation-fenced rollback, and prevention