openedge-replication

# OpenEdge Replication System (Homemade DR) Configures and operates a three-tier database replication architecture for Progress OpenEdge 12.x databases without the OER license — using After-Image shipping, DRBD block replication, ZFS snapshots, keepalived VIP failover, and rfutil roll-forward for sub-second to five-minute RPO/RTO targets. ## TL;DR Checklist - [ ] Select tier based on infrastructure (DRBD+AI for <2s RPO, ZFS+AI for geo-distributed, AI-only rsync as simplest fallback) - [ ] Enable After-Imaging on source with `proutil -C aimage begin` and configure archiver directory - [ ] Deploy AI ship daemon (inotifywait + rsync) on source and AI apply daemon (rfutil roll-forward) on target - [ ] Configure split-brain prevention: lock files, quorum checks, DRBD fence-peer scripts - [ ] Set up keepalived VIP for transparent client failover with health check script - [ ] Validate replication lag stays within RPO thresholds using the health monitor - [ ] Test failover procedure end-to-end before relying on it in production --- ## When to Use Use this skill when: - **Deploying OpenEdge HA without OER** — You need database high availability for Progress OpenEdge 12.x but do not have (or cannot purchase) the OER licensed product - **Designing tiered DR strategy** — You must choose between block-level replication, snapshot-based replication, or AI-only shipping based on RPO/RTO requirements and infrastructure - **Configuring After-Image shipping** — You need to set up continuous transaction-level replication using OpenEdge's built-in AI logging (`proutil -C aimage`) and `rfutil roll-forward` - **Implementing VIP failover** — You want clients to connect to a floating IP that automatically moves between primary and standby on failure (keepalived + VRRP) - **Running a planned or emergency failover** — You need the step-by-step procedures from the runbook for graceful migration or source-server loss scenarios - **Setting up read-only standby** — You want to offload report/BI queries to a standby OpenEdge instance running in `-RO` mode --- ## When NOT to Use Avoid this skill for: - **New database deployments without replication needs** — If you do not need high availability, a single primary with regular backups is simpler - **Sub-millisecond RPO requirements** — This homemade solution cannot match dedicated OER or native clustering products for ultra-low latency failover; use OER instead - **Non-Linux platforms** — DRBD and the provided scripts require Linux kernel modules and systemd; Tier 3 (AI-only) works on any OS but loses BI protection - **Replacing a fully functional OER setup** — If you already have OER replication running, do not migrate to this system without thorough testing - **Database versions before 10.2B** — OPLOCK-based roll-forward requires OpenEdge 10.2B or later; earlier versions need manual `rfutil` apply --- ## Core Workflow ### 1. Assess Requirements and Select Tier Determine which replication tier matches your RPO, RTO, infrastructure, and budget constraints. | Tier | Mechanism | RPO | RTO | BI Protected | Complexity | |------|-----------|-----|-----|-------------|------------| | **Tier 1** | DRBD async + AI shipping | < 2s | < 30s | Yes (DRBD replicates .bi) | Medium | | **Tier 2** | ZFS snapshot + AI shipping | < 5min | < 2min | Yes (in snapshot) | Medium-High | | **Tier 3** | AI-only rsync | 1-5min | < 2min | No | Low | **Checkpoint:** Confirm your RPO and RTO targets, verify available infrastructure (DRBD kernel module, ZFS pool, or neither), and decide if BI protection is required. Tier 3 has no BI protection — uncommitted transactions at crash time are lost. ### 2. Enable After-Imaging on Source Database After-Imaging records committed transaction changes that can be replayed on the target via `rfutil roll-forward`. ```bash # Connect to the source database broker proenv # Enable AI logging (online, no restart required) proutil mydb -C aimage begin # Verify AI is enabled and archiver is running proutil mydb -C aimage list # Expected: After Image Enabled: Yes, Archive Destination: /data/ai-archive, Archive Status: Active # Configure AI archiver directory (copies completed extents for shipping) proutil mydb -C aiarchive enable -aiarcdir /data/ai-archive ``` **Checkpoint:** Verify `Archive Status: Active` in the output. The archiver copies completed AI extents to `/data/ai-archive/`, which is the source for rsync transport. ### 3. Configure Tier-Specific Block/Snapshot Replication #### Tier 1: DRBD Configuration (`/etc/drbd.d/mydb.res`) ``` resource mydb { protocol C; on primary-server { device /dev/drbd1; disk /dev/sda3; address 10.0.0.1:7788; meta-disk internal; } on standby-server { device /dev/drbd1; disk /dev/sda3; address 10.0.0.2:7788; meta-disk internal; } syncer { rate 100M; al-extents 3833; verify-alg sha1; } disk { on-io-error detach; fencing resource-only; } net { cram-hmac-alg sha256; shared-secret "your-secret-here"; after-sb-0p disconnect; after-sb-1p disconnect; after-sb-2p disconnect; } fence-peer { program "/usr/lib/drbd/crm-fence-peer.sh"; } } ``` Initialize: ```bash drbdadm create-md mydb drbdadm -- --overwrite-data-of-primary primary mydb drbdadm secondary mydb # On standby cat /proc/drbd # Verify sync progress ``` **Checkpoint:** DRBD must show `Connected` state and syncing must reach 100% before going live. The `after-sb-2p disconnect` policy prevents split-brain by dropping the connection if both sides are primary. #### Tier 2: ZFS Snapshot Replication ```bash # Source pool setup zpool create -f mirror /dev/sda /dev/sdb -o ashift=12 dbdata zfs set compression=lz4 dbdata zfs set atime=off dbdata zfs set sync=always dbdata/db # Durability for database zfs set checksum=sha256 dbdata/db zfs create dbdata/db zfs create dbdata/ai-archive # Initial full replication zfs send -v dbdata/db | ssh target zfs receive -F dbdata/db # Incremental (run via cron every 5 min) zfs send -v -i dbdata/db@repl-202601011200 dbdata/db@repl-202601011205 \ | ssh target zfs receive -F dbdata/db ``` **Checkpoint:** Verify incremental send completes successfully and ZFS pool has sufficient space for snapshot retention. Use `zfs list -t snapshot -r dbdata/db` to review snapshot history. ### 4. Deploy AI Ship and AI Apply Daemons The AI shipping layer provides transaction-level replication independent of block/snapshot replication, catching transactions between DRBD sync or ZFS snapshot intervals. **Source — AI Ship Daemon** (`/opt/repl/bin/ai-ship-daemon.sh`): ```bash #!/usr/bin/env bash set -euo pipefail SOURCE_DIR="/data/ai-archive" TARGET_HOST="target-server" TARGET_DIR="/data/ai-received" DB_NAME="mydb" LOG_FILE="/var/log/repl/ai-ship.log" PID_FILE="/var/run/repl/ai-ship.pid" running() { kill -0 "$1" 2>/dev/null; } start_daemon() { if [ -f "$PID_FILE" ] && running "$(cat "$PID_FILE")"; then echo "Already running (PID $(cat "$PID_FILE"))" >&2 return 1 fi mkdir -p "$(dirname "$LOG_FILE")" ( while true; do inotifywait -e close_write,moved_to "$SOURCE_DIR" 2>/dev/null || continue sleep 2 latest_ai=$(ls -t "$SOURCE_DIR"/${DB_NAME}.ai* 2>/dev/null | head -1) if [ -n "$latest_ai" ]; then rsync --checksum --compress --timeout=60 "$latest_ai" "$TARGET_HOST:$TARGET_DIR/" \ >> "$LOG_FILE" 2>&1 fi done ) & echo $! > "$PID_FILE" } stop_daemon() { [ -f "$PID_FILE" ] && kill "$(cat "$PID_FILE")" 2>/dev/null || true rm -f "$PID_FILE" } case "${1:-}" in start) start_daemon ;; stop) stop_daemon ;; status) [ -f "$PID_FILE" ] && running "$(cat "$PID_FILE")" && echo "Running" || echo "Stopped" ;; *) echo "Usage: $0 {start|stop|status}"; exit 1 ;; esac ``` **Target — AI Apply Daemon** (`/opt/repl/bin/ai-apply-daemon.sh`): ```sh #!/usr/bin/env bash set -euo pipefail DB_NAME="mydb" RECEIVE_DIR="/data/ai-received" APPLY_DIR="/data/ai-pending" LOG_FILE="/var/log/repl/ai-apply.log" PID_FILE="/var/run/repl/ai-apply.pid" LOCK_FILE="/var/run/repl/ai-apply.lock" APPLY_INTERVAL=60 acquire_lock() { [ -f "$LOCK_FILE" ] && return 1 echo $$ > "$LOCK_FILE" } release_lock() { rm -f "$LOCK_FILE"; } apply_ai() { local pending pending=$(ls -1 "$RECEIVE_DIR"/${DB_NAME}.ai* 2>/dev/null | wc -l) [ "$pending" -eq 0 ] && return 0 ls -1 "$RECEIVE_DIR"/${DB_NAME}.ai* | sort > /tmp/ai-apply-list.txt rfutil "$DB_NAME" -C roll forward oplock -ailist /tmp/ai-apply-list.txt >> "$LOG_FILE" 2>&1 || true rm -f /tmp/ai-apply-list.txt proutil "$DB_NAME" -C aimage begin 2>/dev/null || true for f in "$RECEIVE_DIR"/${DB_NAME}.ai*; do [ -f "$f" ] && mv "$f" "$APPLY_DIR/.applied/" 2>/dev/null || true done } start_daemon() { mkdir -p "$APPLY_DIR/.applied" "$(dirname "$LOG_FILE")" ( while true; do if acquire_lock; then apply_ai; release_lock; fi sleep "$APPLY_INTERVAL" done ) & echo $! > "$PID_FILE" } stop_daemon() { [ -f "$PID_FILE" ] && kill "$(cat "$PID_FILE")" 2>/dev/null || true rm -f "$PID_FILE"; release_lock } case "${1:-}" in start) start_daemon ;; stop) stop_daemon ;; status) [ -f "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null && echo "Running" || echo "Stopped" ;; *) echo "Usage: $0 {start|stop|status}"; exit 1 ;; esac ``` **Checkpoint:** Both daemons must show `Running` status. Verify AI files appear in `$TARGET_DIR/ai-received/` and get applied to `$APPLY_DIR/.applied/`. ### 5. Configure Keepalived VIP (Tier 1 + Optional for All Tiers) The Virtual IP floats between servers, allowing clients to connect without reconnection code on failover. **Keepalived config** (`/etc/keepalived/keepalived.conf`): ``` global_defs { router_id mydb-repl } vrrp_script chk_proserve { script "/opt/repl/bin/check-proserve.sh" interval 5 timeout 3 fall 2 rise 1 } vrrp_instance mydb_vip { state BACKUP interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass mydbsecret } virtual_ipaddress { 192.168.1.100/24 dev eth0 } track_script { chk_proserve } notify /etc/keepalived/notify.sh } ``` **Health check script** (`/opt/repl/bin/check-proserve.sh`): ```bash #!/usr/bin/env bash DB_NAME="${DB_NAME:-mydb}" if proshut "$DB_NAME" -C status 2>/dev/null | grep -q "Multi-User"; then exit 0 else exit 1 fi ``` **Standby startup command:** ```bash # Primary (read-write) proserve mydb -H primary-server -S 3001 # Standby (read-only, for offloading SELECT queries) proserve mydb -RO -H standby-server -S 3001 ``` **Checkpoint:** `ip addr show eth0 | grep 192.168.1.100` must show the VIP on the primary. Stopping the primary should cause the VIP to migrate to standby within ~5 seconds (3 failed checks at 5s interval + fall count). ### 6. Configure systemd Services ```ini # /etc/systemd/system/ai-ship.service [Unit] Description=AI Ship Daemon for OpenEdge Database Replication After=network.target remote-fs.target [Service] Type=forking ExecStart=/opt/repl/bin/ai-ship-daemon.sh start ExecStop=/opt/repl/bin/ai-ship-daemon.sh stop PIDFile=/var/run/repl/ai-ship.pid Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target # /etc/systemd/system/ai-apply.service [Unit] Description=AI Apply Daemon for OpenEdge Database Replication After=network.target remote-fs.target [Service] Type=forking ExecStart=/opt/repl/bin/ai-apply-daemon.sh start ExecStop=/opt/repl/bin/ai-apply-daemon.sh stop PIDFile=/var/run/repl/ai-apply.pid Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target # /etc/systemd/system/keepalived.service [Unit] Description=Keepalived VRRP for OpenEdge VIP Failover After=network.target [Service] Type=forking ExecStart=/usr/sbin/keepalived --vrrp --dump-conf /tmp/keepalived.conf ExecStop=/usr/sbin/keepalived -k PIDFile=/var/run/keepalived.pid [Install] WantedBy=multi-user.target ``` Enable and start: ```bash systemctl daemon-reload systemctl enable --now ai-ship.service ai-apply.service keepalived.service ```