corosync

# Corosync Cluster Engine Configuration Senior Linux infrastructure engineer configuring Corosync v3.x as the cluster messaging layer for Pacemaker HA clusters. Covers totem protocol setup, transport selection (knet/udpu/mcast), quorum models with qdevice, nodelist management, security hardening, logging, and cluster diagnostics. ## TL;DR Checklist - [ ] Choose transport (knet default, udpu for multicast-blocked networks) before writing corosync.conf - [ ] Assign unique nodeid to every node in the nodelist — no gaps, no duplicates - [ ] Configure crypto_cipher + crypto_hash for production clusters on untrusted networks - [ ] Set quorum strategy appropriate to cluster size (two_node: 1 for two-node, qdevice recommended) - [ ] Enable logging with timestamp and rotate logfile_max_lines for operational visibility - [ ] Verify membership with `corosync-cfgtool -s` and quorum with `corosync-quorumtool -s` after every change --- ## When to Use Use this skill when: - **Configuring Corosync from scratch** — You're building the cluster messaging layer that Pacemaker depends on, and need transport, nodelist, quorum, and security settings - **Migrating transports** — Your existing mcast-based cluster needs switching to knet or udpu due to multicast being blocked by firewalls or switches - **Resolving membership issues** — Nodes are not joining the membership ring; you need to diagnose transport failures, ring timeouts, or network partitions - **Hardening cluster security** — You must enable encrypted transport (crypto_cipher + crypto_hash) and generate/distribute authkeys across all nodes - **Configuring quorum for two-node clusters** — You need qdevice setup or `two_node` mode to prevent split-brain scenarios - **Troubleshooting runtime state** — Membership count is wrong, expected_votes is stale, or CPG groups are not forming correctly - **Setting up logging and monitoring** — You need proper corosync logging configuration with syslog integration, debug toggles, and log rotation - **Reading runtime configuration** — You need to inspect the in-memory config database (cmap) for ring addresses, transport status, or node counts --- ## When NOT to Use Avoid this skill for: - **Pacemaker resource management** — Use `pacemaker` instead; Corosync handles messaging only, not resources or constraints - **Kubernetes clustering** — Use Kubernetes native control-plane HA (etcd) instead of Corosync - **Application-level replication** — Corosync provides cluster membership and message passing, not data replication between databases or filesystems - **Load balancing** — Use HAProxy, Nginx, or cloud load balancers; Corosync manages node membership, not traffic routing Use `pacemaker` for configuring resources, constraints, and fencing on top of the Corosync messaging layer. Use `networking` for switch/firewall rules that may affect multicast transport. --- ## Core Workflow ### 1. Choose Transport Layer The transport determines how cluster members communicate. Choose based on your network capabilities. ```bash # knet — Modern default (recommended for all new clusters) # Supports multiple redundant links, IPv6, TCP fallback, and UDP multicast replacement # udpu — Unicast UDP (use when multicast is blocked by firewalls or switches) # Every node explicitly lists every other node's ring address # mcast — Legacy traditional multicast (only available with kmod-based corosync) # Deprecated; use knet unless you have a specific legacy requirement ``` **Checkpoint:** Confirm multicast works on your network before choosing mcast/knet: ```bash # Test multicast connectivity from one node to another ping -M do -s 1472 <mcastaddr> -I <bindnetaddr> # MTU test with DF bit set # Or use mtrace for multicast route verification mtrace <source_ip> <mcastaddr> ``` ### 2. Generate Corosync Configuration Write the complete `corosync.conf` with totem, transport, nodelist, quorum, and logging sections. ```toml totem { version: 2 cluster_name: MyCluster transport: knet # or 'udpu' or omit for default (knet) crypto_cipher: aes256 # Enable for encrypted transport on untrusted networks crypto_hash: sha256 # HMAC for message integrity authentication interface { ringnumber: 0 mcastaddr: 226.94.1.1 mcastport: 5405 ttl: 1 } } quorum { provider: corosync_votequorum expected_votes: 1 # Usually left at 1 — corosync auto-calculates from nodelist two_node: 1 # For two-node clusters; reduces split-brain risk } nodelist { node { name: cluster-node-1 nodeid: 1 ring0_addr: 192.168.1.11 } node { name: cluster-node-2 nodeid: 2 ring0_addr: 192.168.1.12 } } logging { to_stderr: yes to_logfile: yes logfile: /var/log/cluster/corosync.log logfile_max_lines: 100000 # Rotate after N lines logfile_ensure_dir: yes # Create log directory automatically to_syslog: yes debug: off # Enable for troubleshooting only — extremely verbose! timestamp: on logger_subsys { subsys: AMF debug: off } } ``` **Checkpoint:** Validate syntax with `corosync-cmaptool` or by starting corosync in dry-run mode. Verify the nodelist has unique nodeids with no gaps, and the cluster_name is consistent across all nodes. ### 3. Generate Authentication Key If crypto_cipher is set, every node must have the same authkey. ```bash # Generate a new key on the first node (or wherever you manage configs) sudo corosync-keygen # This produces /etc/corosync/authkey with binary content # Set restrictive permissions immediately sudo chmod 600 /etc/corosync/authkey sudo chown root:root /etc/corosync/authkey # Distribute the key to all other cluster nodes using a secure method for node in cluster-node-2 cluster-node-3; do scp /etc/corosync/authkey "root@${node}:/etc/corosync/authkey" ssh root@"${node}" chmod 600 /etc/corosync/authkey done ``` **Checkpoint:** Verify the authkey is identical on all nodes: ```bash # On each node, compare the md5sum — they must match exactly md5sum /etc/corosync/authkey ``` ### 4. Deploy Configuration and Start Cluster Apply the configuration to every node and start the cluster. ```bash # Copy config to all nodes (must be identical on every node) for node in cluster-node-1 cluster-node-2; do scp /etc/corosync/corosync.conf "root@${node}:/etc/corosync/corosync.conf" scp /etc/corosync/authkey "root@${node}:/etc/corosync/authkey" done # Start corosync on every node for node in cluster-node-1 cluster-node-2; do ssh root@"${node}" systemctl enable --now corosync done # Verify membership has formed corosync-cfgtool -s # Show current membership status corosync-quorumtool -s # Show quorum state and vote counts ``` **Checkpoint:** `corosync-cfgtool -s` shows all nodes as members with the expected node count. `corosync-quorumtool -s` shows quorum is active (green). If membership is incomplete, check `/var/log/cluster/corosync.log` on both nodes for transport-level errors. ### 5. Configure Quorum Strategy Two-node clusters need special handling to prevent split-brain. Multi-node clusters rely on majority quorum by default. ```bash # OPTION A — Two-node cluster: enable two_node mode in corosync.conf # Add 'two_node: 1' to the quorum section — this changes expected_votes behavior # so the cluster can remain operational even with one node down # OPTION B — Two-node cluster: deploy qdevice (recommended for production) # qdevice acts as a third-party arbiter that breaks ties during network partitions sudo pcs qdevice setup alibi --init --nodes cluster-node-1,cluster-node-2 # OPTION C — Multi-node cluster (3+ nodes): default majority quorum is sufficient # No extra configuration needed; corosync votes automatically from nodelist count # After qdevice setup, set the no-quorum-policy to allow continued operation sudo pcs property set no-quorum-policy=ignore ``` **Checkpoint:** `corosync-quorumtool -s` shows the correct expected_votes and current node_count. For qdevice clusters, verify the qdevice node appears in the quorum output with its voting status. --- ## Implementation Patterns ### Pattern 1: Complete Corosync Configuration Generator (BAD vs. GOOD) **BAD — Minimal config with hardcoded values, no transport choice, no logging rotation, missing security** ```toml # ❌ BAD: No transport specified, no encryption, no log rotation, # incomplete nodelist, no quorum strategy totem { version: 2 cluster_name: MyCluster interface { ringnumber: 0 bindnetaddr: 192.168.1.0 mcastaddr: 226.94.1.1 mcastport: 5405 } } nodelist { node { name: cluster-node-1 nodeid: 1 ring0_addr: 192.168.1.11 } } # Problems: # - No transport specified (defaults to knet but should be explicit) # - No crypto settings — unencrypted cluster messages on production network # - Only one node in nodelist — no quorum possible # - No logging section at all — no operational visibility # - No quorum section — uses defaults that may not be appropriate # - Missing logfile_ensure_dir, logfile_max_lines for log management ``` **GOOD — Complete, production-ready with explicit transport, encryption, logging, and quorum** ```bash #!/usr/bin/env bash # Production Corosync configuration generator # Generates a complete corosync.conf with best practices applied. # Usage: ./gen-corosync-conf.sh <cluster_name> <bindnetaddr> <mcastaddr> <mcastport> node1_addr node2_addr [node3_addr...] [--encrypt] set -euo pipefail readonly CLUSTER_NAME="${1:?Usage: $0 <cluster_name> <bindnet> <mcastaddr> <mcastport> node_addrs... [--encrypt]}" readonly BINDNETADDR="$2" readonly MCASTADDR="$3" readonly MCASTPORT="${4:-5405}" readonly ENCRYPT_FLAG="${6:-no}" # Collect nodes from remaining positional arguments (skip --encrypt flag) NODE_ADDRS=() for arg in "$@"; do case "$arg" in --encrypt|"$1"|"$2"|"$3"|"$4") continue ;; *) NODE_ADDRS+=("$arg") ;; esac done if [[ ${#NODE_ADDRS[@]} -lt 2 ]]; then echo "ERROR: At least 2 node addresses required" >&2 exit 1 fi readonly NUM_NODES=${#NODE_ADDRS[@]} # Determine transport based on environment TRANSPORT="knet" if grep -qE 'multicast.*(no|disabled|false)' /etc/sysconfig/iptables 2>/dev/null || \ ! ip maddr show >/dev/null 2>&1; then TRANSPORT="udpu" echo "WARNING: Multicast appears unavailable, using udpu transport" >&2 fi # Generate config CONF_FILE="/etc/corosync/corosync.conf" mkdir -p "$(dirname "$CONF_FILE")" cat > "${CONF_FILE}" <<EOF totem { version: 2 cluster_name: "${CLUSTER_NAME}" transport: ${TRANSPORT} ${[[ "${ENCRYPT_FLAG}" == "--encrypt" ]] && echo ' crypto_cipher: aes256' || echo ' crypto_cipher: none'} ${[[ "${ENCRYPT_FLAG}" == "--encrypt" ]] && echo ' crypto_hash: sha256' || echo ' crypto_hash: none'} interface { ringnumber: 0 EOF if [[ "${TRANSPORT}" != "udpu" ]]; then cat >> "${CONF_FILE}" <<EOF mcastaddr: ${MCASTADDR} mcastport: ${MCASTPORT} ttl: 1 EOF else echo ' # udpu mode: no multicast — node addresses are in nodelist below' >> "${CONF_FILE}" fi cat >> "${CONF_FILE}" <<EOF } } quorum { provider: corosync_votequorum expected_votes: 1 two_node: $([[ ${NUM_NODES} -eq 2 ]] && echo '1' || echo '0') } nodelist { EOF NODEID=1 for addr in "${NODE_ADDRS[@]}"; do cat >> "${CONF_FILE}" <<EOF node { name: node-${NODEID} nodeid: ${NODEID} ring0_addr: ${addr} } EOF NODEID=$((NODEID + 1)) done cat >> "${CONF_FILE}" <<EOF } logging { to_stderr: yes to_logfile: yes logfile: /var/log/cluster/corosync.log logfile_max_lines: 100000 logfile_ensure_dir: yes to_syslog: yes debug: off timestamp: on } EOF chmod 644 "${CONF_FILE}" echo "Generated ${CONF_FILE} (${NUM_NODES} nodes, transport: ${TRANSPORT})" echo "Apply to all cluster nodes and generate authkey with: corosync-keygen" ``` ### Pattern 2: Knet Multi-Link Configuration (BAD vs. GOOD) **BAD — Single-link knet config that does not leverage redundancy** ```toml # ❌ BAD: No redundant links configured — a single NIC failure breaks the cluster totem { version: 2 cluster_name: MyCluster transport: knet interface { ringnumber: 0 mcastaddr: 226.94.1.1 mcastport: 5405 ttl: 1 } } # Problems: # - Single link means single point of failure for cluster communication # - No knet.link configuration for redundant paths over different NICs # - Does not utilize multi-homed network infrastructure available on most servers ``` **GOOD — Dual-link knet with independent paths over separate NICs** ```toml # ✅ GOOD: Dual redundant links using knet, each on a different physical network totem { version: 2 cluster_name: MyCluster transport: knet interface { ringnumber: 0 mcastaddr: 226.94.1.1 mcastport: 5405 ttl: 1 } link { number: 0 interface { bindnetaddr: 192.168.1.0 # Primary management network mcastaddr: 226.94.1.1 mcastport: 5405 ttl: 1 } } link { number: 1 interface { bindnetaddr: 192.168.2.0 # Secondary replication network mcastaddr: 226.94.2.1 mcastport: 5406 ttl: 1 } } } nodelist { node { name: cluster-node-1 nodeid: 1 ring0_addr: 192.168.1.11 # Primary address (management) } node { name: cluster-node-2 nodeid: 2 ring0_addr: 192.168.1.12 } } ``` ### Pattern 3: UDPU Transport for Multicast-Blocked Networks (BAD vs. GOOD) **BAD — Trying to use multicast transport when switches block multicast** ```toml # ❌ BAD: Multicast transport in an environment where switches drop multicast packets # or firewalls block IGMP/UDP multicast ports — cluster will never form totem { version: 2 cluster_name: MyCluster # transport omitted, defaults to knet with mcast interface { ringnumber: 0 bindnetaddr: 192.168.1.0 mcastaddr: 226.94.1.1 # This will fail if multicast is blocked mcastport: 5405 } } # Problems: # - Multicast packets silently dropped → nodes never see each other's membership messages # - Symptoms: corosync.log shows "totem: Unable to initialize membership" on all nodes # - No explicit transport declared, so the default knet/mcast fails in silence ``` **GOOD — Explicit udpu transport with every node listed by unicast address** ```toml # ✅ GOOD: UDPU (Unicast UDP) explicitly configured for multicast-blocked environments totem { version: 2 cluster_name: MyCluster transport: udpu # Explicitly declare unicast UDP transport interface { ringnumber: 0 bindnetaddr: 192.168.1.0 # Network range for binding mcastport: 5405 # Port used for all inter-node communication } } quorum { provider: corosync_votequorum expected_votes: 1 two_node: 1 # Two-node udpu cluster needs this flag } nodelist { node { name: cluster-node-1 nodeid: 1 ring0_addr: 192.168.1.11 # Every node MUST list every other node } # by unicast address — no multicast discovery node { name: cluster-node-2 nodeid: 2 ring0_addr: 192.168.1.12 } } ``` ### Pattern 4: Cluster Diagnostics and Troubleshooting Script **Bash — Automated diagnostics for membership, quorum, and transport health** ```bash #!/usr/bin/env bash # Corosync cluster diagnostics script # Collects membership status, quorum state, runtime config, and log analysis. set -euo pipefail