pacemaker

# Pacemaker HA Cluster Management Senior Linux infrastructure engineer managing Pacemaker 3.x HA clusters with PCS and crmsh for resource provisioning, constraints, STONITH fencing, quorum configuration, and cluster lifecycle operations. ## TL;DR Checklist - [ ] Verify node connectivity and corosync membership before creating any resources - [ ] Configure STONITH (fencing) devices on every cluster — never disable in production - [ ] Set up quorum strategy (qdevice for two-node clusters) before adding workloads - [ ] Create resources with explicit monitor operations and appropriate intervals - [ ] Define colocation AND ordering constraints — they are independent, not interchangeable - [ ] Use `pcs resource cleanup` after node recovery to clear stale failcounts --- ## When to Use Use this skill when: - **Building HA clusters** — You're provisioning a two-node or multi-node Pacemaker cluster with Corosync and need end-to-end setup guidance - **Managing cluster resources** — You need to create, migrate, stop, start, or troubleshoot OCF resource agents (VirtualIP, Apache, database, etc.) - **Configuring fencing** — You must set up STONITH/fence devices (IPMI, iLO, DRAC, WTI) for a production cluster - **Fixing split-brain risk** — A two-node cluster needs qdevice or corosync `two_node` quorum configuration - **Troubleshooting clusters** — Resources won't start, nodes show uncertain state, or CIB conflicts need resolution - **Implementing master/slave patterns** — You're deploying promotable clones for databases (PostgreSQL, MySQL), DRBD, or other active/passive workloads - **Running cluster diagnostics** — You need to generate reports with `crm_report` or analyze CIB state --- ## When NOT to Use Avoid this skill for: - **Kubernetes clustering** — Use Kubernetes native HA patterns instead; Pacemaker is not a container orchestrator - **Load balancing** — Use HAProxy, Nginx, or cloud load balancers; Pacemaker manages node failover, not traffic distribution - **Single-node high availability** — If only one server exists, you need backups/replication, not a cluster - **Application-level redundancy** — Design your application to be stateless and multi-instance; Pacemaker is infrastructure-level failover Use `linux-services` for systemd unit configuration that complements OCF resource agents. Use `corosync` for network-layer membership and token configuration. --- ## Core Workflow ### 1. Cluster Initialization and Node Setup Create the cluster from scratch with Corosync as the messaging layer. ```bash # On BOTH nodes — install packages (RHEL/Rocky/Alma) sudo dnf install -y pcs pacemaker corosync # Enable and start PCS daemon on both nodes sudo systemctl enable --now pcsd # Set identical hacluster password on every node echo 'changeme' | sudo passwd hacluster --stdin # Authenticate PCS between nodes from the first node sudo pcs cluster auth node1 node2 -u hacluster -p 'changeme' --force # Create and start the cluster sudo pcs cluster setup --name mycluster node1 node2 --start --enable ``` **Checkpoint:** Run `pcs status` — all nodes should show as "Online" and the cluster state should be "Active." If any node is "Offline," check corosync logs: `journalctl -u corosync --since "5 minutes ago"`. ### 2. Configure STONITH (Fencing) Devices Every production cluster MUST have fencing configured before adding workloads. ```bash # Example: IPMI fence device for each node sudo pcs stonith create fence-node1-ipmi \ stonith:fence_ipmilan \ ipaddr="192.168.1.100" \ login="admin" \ passwd="secret" \ pcmk_host_list="node1" \ power_wait="5" \ op monitor interval="60s" sudo pcs stonith create fence-node2-ipmi \ stonith:fence_ipmilan \ ipaddr="192.168.1.101" \ login="admin" \ passwd="secret" \ pcmk_host_list="node2" \ power_wait="5" \ op monitor interval="60s" # Enable fencing on the cluster sudo pcs property set stonith-enabled=true # Set fencing delay to prevent race conditions during multi-node outages sudo pcs property set pcmk_delay_base=30 sudo pcs property set pcmk_delay_max=60 # Test each fence device BEFORE declaring production readiness sudo pcs stonith fence node1 --force sudo pcs stonith fence node2 --force ``` **Checkpoint:** Both fence devices show as "Started" in `pcs status resources`. Each test fence action successfully powers cycles the target node (or reports success from the management interface). ### 3. Configure Quorum Strategy Two-node clusters without qdevice are susceptible to split-brain. Choose a strategy: ```bash # OPTION A — Recommended for two-node clusters: qdevice (RHEL 9+) sudo pcs qdevice setup alibi --init --nodes node1,node2 # OPTION B — Corosync-level two_node mode (works without qdevice) # Edit corosync.conf on BOTH nodes: # quorum { expected_votes: 2; two_node: 1 } # For multi-node clusters (3+), the default majority-quorum policy works fine # sudo pcs property set no-quorum-policy=stop # default — stops all resources # Set safe no-quorum policy for qdevice-enabled clusters sudo pcs property set no-quorum-policy=ignore ``` **Checkpoint:** `pcs status corosync` shows correct membership count. `pcs status quorum` confirms expected votes match and the node count provides adequate quorum. ### 4. Create Resources with Constraints Define virtual IP, application, and storage resources with proper ordering. ```bash # Create a managed VirtualIP resource sudo pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \ ip="192.168.1.200" \ cidr_netmask="24" \ op monitor interval="30s" # Create an Apache resource (runs on the same node as VirtualIP) sudo pcs resource create Apache ocf:heartbeat:apache \ configfile="/etc/httpd/conf/httpd.conf" \ statusurl="http://localhost/server-status" \ op monitor interval="60s" \ op start timeout="60s" \ op stop timeout="60s" # Create a web application resource (managed by Apache) sudo pcs resource create WebSite ocf:heartbeat:apache \ configfile="/etc/httpd/conf.d/webapp.conf" \ op monitor interval="30s" # Colocation: VirtualIP and Apache must be on the same node sudo pcs constraint colocation add apache-with-vip VirtualIP Apache INFINITY # Ordering: VirtualIP starts before Apache sudo pcs constraint order VirtualIP then Apache kind=Mandatory # Resource group alternative (bundles both colocation AND ordering) # sudo pcs resource group add web-stack VirtualIP Apache WebSite ``` **Checkpoint:** `pcs status resources` shows all three resources running on the same node. Verify with `crm_simulate -s` that the constraint graph is consistent. ### 5. Implement Promotable Clones (Master/Slave) For databases and shared-storage workloads requiring active/passive failover. ```bash # Create a DRBD resource with promote operation sudo pcs resource create db-storage ocf:linbit:drbd \ drbd_resource=db0 \ op monitor interval="30s" role=Slave \ op promote interval="0s" role=Master # Promote it to master/slave clone sudo pcs resource master db-master db-storage \ master-max=1 \ master-node-max=1 \ clone-max=2 \ clone-node-max=1 \ notify=true # Ensure application runs only on the current master sudo pcs constraint colocation add db-app-with-db-master WebSite with db-master INFINITY sudo pcs constraint order db-master then WebSite kind=Mandatory ``` **Checkpoint:** `pcs status resources` shows one node as Master and the other as Slave for `db-master`. Failover test: `pcs cluster standby node1` — verify that the database promotes on node2 within the expected window. --- ## Implementation Patterns ### Pattern 1: Complete Cluster Build Script (BAD vs. GOOD) **BAD — Manual step-by-step without error handling, skipping fencing, no idempotency** ```bash # ❌ BAD: No error handling, skips critical security steps, not repeatable sudo systemctl enable pcsd sudo systemctl start pcsd echo "changeme" | passwd hacluster --stdin pcs cluster setup mycluster node1 node2 pcs property set stonith-enabled=false # ← NEVER DISABLE STONITH pcs resource create VIP ocf:heartbeat:IPaddr2 ip=10.0.0.50 op monitor interval=30s ``` **Problems:** - No `set -euo pipefail` — failures are silently ignored - Disables STONITH — Red Hat does NOT support unfenced clusters; split-brain data corruption risk - Hardcoded passwords in plain text without secure handling - No authentication between nodes (pcs cluster auth missing) - No quorum configuration for the cluster type - No start/stop timeout definitions on resources - Not idempotent — re-running will create duplicate resources or fail **GOOD — Idempotent, secure, production-ready with fencing and quorum** ```bash #!/usr/bin/env bash # Production-ready Pacemaker cluster setup script # Usage: ./setup-cluster.sh <cluster_name> node1 node2 [node3...] [--qdevice-node <third_node>] set -euo pipefail readonly CLUSTER_NAME="${1:?Usage: $0 <cluster_name> node1 node2 ...}" readonly NODES=("${@:2}") readonly QDEVICE_NODE="${4:-}" # Optional third node for qdevice # ─── Validation ────────────────────────────────────────────── if [[ ${#NODES[@]} -lt 2 ]]; then echo "ERROR: At least 2 nodes required" >&2 exit 1 fi echo "=== Pacemaker Cluster Setup: ${CLUSTER_NAME} ===" echo "Nodes: ${NODES[*]}" # ─── Step 1: Package Installation (idempotent) ────────────── echo "[1/6] Installing packages..." for node in "${NODES[@]}"; do echo " → Ensuring packages on ${node}..." ssh -o StrictHostKeyChecking=no root@"${node}" \ dnf install -y --setopt=install_weak_deps=false \ pcs pacemaker corosync || { echo "ERROR: Failed to install packages on ${node}" >&2 exit 1 } ssh root@"${node}" systemctl enable --now pcsd done # ─── Step 2: Authentication ───────────────────────────────── echo "[2/6] Configuring node authentication..." for node in "${NODES[@]}"; do # Check if already authenticated by attempting a no-op command if ! ssh root@"${node}" pcs cluster auth >/dev/null 2>&1; then echo " → Authenticating with ${node}..." ssh root@"${node}" passwd hacluster --stdin <<< "${HA_CLUSTER_PASSWORD:-changeme}" fi done # ─── Step 3: Cluster Creation ─────────────────────────────── echo "[3/6] Creating cluster: ${CLUSTER_NAME}..." if ! pcs cluster auth --with-force "${NODES[@]}" -u hacluster \ -p "${HA_CLUSTER_PASSWORD:-changeme}" >/dev/null 2>&1; then echo "ERROR: Node authentication failed. Verify hacluster password matches." >&2 exit 1 fi if ! pcs cluster setup --name "${CLUSTER_NAME}" "${NODES[@]}" \ --start --enable --with-force; then echo "ERROR: Cluster setup failed" >&2 exit 1 fi # ─── Step 4: STONITH Configuration ────────────────────────── echo "[4/6] Configuring fencing..." for i in "${!NODES[@]}"; do local node="${NODES[$i]}" local ipmi_addr="${IPMI_ADDRESSES[$i]:-}" if [[ -n "${ipmi_addr}" ]]; then echo " → Creating fence device for ${node} (IPMI: ${ipmi_addr})" pcs stonith create "fence-${node}" \ stonith:fence_ipmilan \ ipaddr="${ipmi_addr}" \ login="${IPMI_LOGIN:-admin}" \ passwd="${IPMI_PASS:-changeme}" \ pcmk_host_list="${node}" \ power_wait="5" \ op monitor interval="60s" || true fi done pcs property set stonith-enabled=true 2>/dev/null || true pcs property set pcmk_delay_base=30 2>/dev/null || true pcs property set pcmk_delay_max=60 2>/dev/null || true # ─── Step 5: Quorum Configuration ─────────────────────────── echo "[5/6] Configuring quorum..." if [[ ${#NODES[@]} -eq 2 ]] && [[ -z "${QDEVICE_NODE}" ]]; then echo " WARNING: Two-node cluster without qdevice — split-brain risk" echo " Apply corosync two_node=1 in quorum configuration." pcs property set no-quorum-policy=stop 2>/dev/null || true elif [[ -n "${QDEVICE_NODE}" ]]; then echo " → Setting up qdevice on ${QDEVICE_NODE}" pcs qdevice setup alibi --init --nodes "${NODES[*]}" 2>/dev/null || true pcs property set no-quorum-policy=ignore 2>/dev/null || true fi # ─── Step 6: Verification ─────────────────────────────────── echo "[6/6] Verifying cluster state..." sleep 5 if ! pcs cluster status >/dev/null 2>&1; then echo "ERROR: Cluster is not responding. Check corosync and pacemaker logs." >&2 exit 1 fi echo "" echo "=== Cluster ${CLUSTER_NAME} Ready ===" pcs status nodes || true pcs stonith list || true pcs property show stonith-enabled no-quorum-policy || true ``` ### Pattern 2: Constraint Configuration (BAD vs. GOOD) **BAD — Using resource groups when specific control is needed, missing ordering** ```bash # ❌ BAD: Resource group forces implicit ordering you may not want, # and makes per-resource meta-options harder to manage sudo pcs resource group add web-stack VirtualIP Apache WebSite # Problem: You cannot set different monitor intervals or failure actions # for individual resources within a group. The group acts as one unit. # ❌ BAD: Colocation without ordering — resources run together but may start in wrong order sudo pcs constraint colocation add apache-with-vip VirtualIP Apache INFINITY # Problem: Apache might attempt to start before the VIP exists, # causing bind failures even though they end up on the same node. ``` **GOOD — Explicit colocation AND ordering with resource-level options** ```bash # ✅ GOOD: Separate resources with explicit constraints gives full control # over per-resource monitor intervals, failure actions, and meta-options # Create resources independently with tuned operations sudo pcs resource create VirtualIP ocf:heartbeat:IPaddr2 \ ip="192.168.1.200" \ cidr_netmask="24" \ op monitor interval="30s" \ meta failure-timeout="90s" sudo pcs resource create Apache ocf:heartbeat:apache \ configfile="/etc/httpd/conf/httpd.conf" \ op monitor interval="60s" timeout="60s" \ op start timeout="60s" \ op stop timeout="60s" \ meta failure-timeout="90s" # Colocation constraint: run together on the same node sudo pcs constraint colocation add apache-with-vip \ VirtualIP Apache INFINITY # Ordering constraint: define startup sequence explicitly sudo pcs constraint order VirtualIP then Apache \ kind=Mandatory score=INFINITY # Location preference: prefer this node for VIP (for controlled failover) sudo pcs constraint location VirtualIP prefers node1 INFINITY # Verify the full constraint graph pcs constraint list --full crm_simulate -s # Simulate to verify no conflicts ``` ### Pattern 3: Cluster Diagnostic and Recovery Script **Bash — Automated cluster diagnostics with safe recovery** ```bash #!/usr/bin/env bash # Pacemaker cluster diagnostic and recovery script # Generates comprehensive diagnostics and offers safe recovery actions. set -euo pipefail readonly REPORT_DIR="/var/tmp/pacemaker-diagnostics" readonly TIMESTAMP="$(date +%Y%m%d-%H%M%S)" mkdir -p "${REPORT_DIR}/${TIMESTAMP}" echo "=== Pacemaker Cluster Diagnostics ===" echo "Timestamp: ${TIMESTAMP}" # ─── Collect diagnostic data ──────────────────────────────── collect() { local label="$1"; shift echo "[+] Collecting: ${label}" "$@" > "${REPORT_DIR}/${TIMESTAMP}/${label}.log" 2>&1 || true } collect "nodes_status" pcs status nodes collect "full_status" pcs status collect "resources_status" pcs status resources collect "corosync_status" pcs status corosync collect "quorum_status" pcs status quorum collect "stonith_list" pcs stonith list collect "stonith_config" pcs stonith config 2>/dev/null || true collect "constraints" pcs constraint list --full collect "properties" pcs property show collect "cib_dump" pcs cluster cib 2>/dev/null || true # ─── Generate crm_report (if available) ───────────────────── if command -v crm_report &>/dev/null; then