deploy-spur

name

deploy-spur

description

Use when the user asks to deploy/install a Spur cluster, or roll out a new build to an already-running one, on one or more bare-metal hosts over SSH. Covers every topology the Ansible playbook does — single-node, multi-node, HA (multi-controller Raft), and HA with separate compute nodes — plus optional PostgreSQL accounting (embedded in spurctld) and WireGuard mesh. Installs daemons as systemd services and Slurm-compatible CLI symlinks. Drives everything with plain SSH + bash; no Ansible required. ALWAYS asks the user up-front for mode + topology + controller count + accounting before touching anything.

Deploy Spur Cluster

Spur is an AI-native job scheduler with these daemons:

• spurctld — controller / scheduler / Raft consensus (1 instance, or ≥ 3 for HA). Also serves accounting (sacct/fairshare, backed by PostgreSQL) in-process on its own gRPC port whenever [accounting].database_url is set — there is no separate accounting daemon. Only Postgres itself is a distinct service, on ACCT_HOST (default: first controller; may be a dedicated node).
• spurd — node agent, runs on every compute host

Pre-merge Spur builds additionally shipped a standalone spurdbd accounting daemon; upstream folded it into spurctld. If you're upgrading a cluster that still has spurdbd.service active, see Step 5b: migrating off a standalone spurdbd below — don't skip it, or you'll end up with two accounting paths fighting over the same Postgres.

This skill stands a cluster up with only SSH + bash on the targets. It is the standalone equivalent of ansible/ and reaches the same end state: daemons run as systemd services (survive reboot), Slurm-compatible CLI names are symlinked, and accounting is optional (default on). One flow covers all four topologies — only the host list, Raft topology, and which hosts run an agent differ. A rolling-upgrade flow (Step 12) upgrades an already-running cluster one host at a time instead of bouncing everything at once.

Defaults (override only if the user asks):

Var	Default
SPUR_HOME	/root/spur
SPUR_INSTALL_DIR	/root/.local/bin
SPUR_VERSION	latest (passed to install.sh; or nightly / vX.Y.Z)
SPUR_BINARY_SRC	(empty) — local dir with pre-built spur/spurctld/spurd and optionally spur_mpi_pmix.so; used when set, else install.sh
SPUR_MPI_PLUGIN_DIR	/usr/lib/spur — where spur_mpi_pmix.so is installed on agents (matches spurd default)
SPUR_CONTROLLER_PORT	6817
SPUR_AGENT_PORT	6818
SPUR_RAFT_PORT	6821 (hardcoded inside spurctld; cannot be changed via CLI)
ACCT_DB_PORT	5432 (PostgreSQL)
SPUR_CLUSTER_NAME	spur-cluster
SPUR_LOG_LEVEL	info
SPUR_WIPE_STATE	false (preserve Raft state so re-runs/upgrades are non-destructive; set true for a fresh install or intentional Raft reinit)
ACCOUNTING	true (deploy PostgreSQL; accounting is served by spurctld itself; set false to skip)
ACCT_DB_NAME / ACCT_DB_USER / ACCT_DB_PASSWORD	spur / spur / spur
TRANSPORT	direct (LAN) — or wireguard for an encrypted mesh
ROLLING_BATCH_SIZE	1 (agents upgraded per batch in Step 12; controllers are always one at a time)
SSH user	root (unless the user specifies otherwise)

Every command below is written to work whether the SSH user is root or a non-root user with (passwordless) sudo — confirm sudo -n true succeeds during Step 1's preflight. SPUR_HOME/SPUR_INSTALL_DIR default under /root, which is mode 0700: a non-root user cannot even cd/execute/test -x into it, let alone write there, and no chown of a subdirectory fixes this (the block is on traversing /root itself). So:

• Every remote command that reads/writes under /root, /etc/systemd/system, or runs spur/sbatch/sacct/etc. must be sudo-prefixed — not just the file-writing steps. This applies uniformly (as root, sudo is a harmless no-op).
• scp and heredoc redirects (cat > /path <<EOF) run as the plain SSH user and cannot land a file directly under /root even with the SSH user later sudo-reading it. Copy to /tmp first, then sudo install/sudo mv it into place.
• Alternatively, set SPUR_INSTALL_DIR/SPUR_HOME to a world-traversable path (e.g. /opt/spur) up front to sidestep all of this — but then every install/sudo note below is still harmless, just unnecessary.

Password-based SSH works too — every ssh/scp command in this skill is a plain invocation with no auth-method assumptions baked in, so if key-based auth isn't set up, prefix each one with sshpass -p "$SSH_PASSWORD" ssh -o StrictHostKeyChecking=no ... (and the scp equivalent). Do not add -o BatchMode=yes anywhere — it disables SSH's password prompt outright and silently breaks password auth even with sshpass supplying the answer.

Step 0: gather inputs (MANDATORY — do not skip)

Before any SSH, ask the user (use AskUserQuestion for anything they didn't state; don't guess):

1. Deployment mode — pick exactly one:

   Mode	Use when
   single-node	one host runs controller and agent
   multi-node	1 controller, N compute agents (controller may also run an agent — hyperconverged)
   ha	≥ 3 controllers (Raft), N agents; controllers may be hyperconverged or dedicated (separate compute)

2. Hosts — for each role:

   • CONTROLLERS — SSH targets running spurctld. Counts: single-node 1, multi-node 1, ha odd ≥ 3.
   • AGENTS — SSH targets running spurd. Any number ≥ 1. A host may appear in both lists (hyperconverged) or only in AGENTS (dedicated compute / "separate compute" HA).

3. Accounting — deploy PostgreSQL for sacct/fairshare (served by spurctld itself, no separate daemon)? Default yes. If no, set ACCOUNTING=false; job submission still works, only sacct is unavailable.

4. Transport — direct (LAN, default) or wireguard (encrypted mesh). WireGuard adds Step 2b; everything else is identical (config advertises WG IPs instead of LAN IPs).

For HA, warn the user if controller count is even or < 3:

• N=1 → not HA; suggest multi-node.
• N=2 → "zero fault tolerance" (quorum 2, tolerates 0 failures) — code-path testing only.
• even N ≥ 4 → suggest N−1 (strictly better).

Topologies map to inventory shape exactly like the playbook:

• single-node → same host in CONTROLLERS and AGENTS
• multi-node → 1 controller, N agents
• HA hyperconverged → controllers also in AGENTS
• HA + separate compute → controllers not in AGENTS; distinct agent hosts

Once gathered, define the arrays the rest of the skill uses:

CONTROLLERS=( user@host1 user@host2 user@host3 )   # ordered — index = Raft node_id - 1; ORDER MUST BE STABLE
AGENTS=(     user@host4 )                            # may overlap CONTROLLERS (hyperconverged) or be disjoint
LOGIN=(      )                                        # OPTIONAL dedicated submission/login nodes: CLI only, no daemon. Empty = none.
SSH_USER=root
TRANSPORT=direct                                     # or wireguard
ACCOUNTING=true                                      # or false
ACCT_HOST="${CONTROLLERS[0]}"                         # accounting host: default first controller; may be ANY host — a controller, an agent, or a dedicated node (add it to HOSTS_ALL if dedicated)

SPUR_HOME=/root/spur
SPUR_INSTALL_DIR=/root/.local/bin
SPUR_VERSION=latest
SPUR_BINARY_SRC=                                     # e.g. /tmp/spur-bin to push pre-built binaries
SPUR_MPI_PLUGIN_DIR=/usr/lib/spur
SPUR_CONTROLLER_PORT=6817; SPUR_AGENT_PORT=6818; SPUR_RAFT_PORT=6821; ACCT_DB_PORT=5432
SPUR_CLUSTER_NAME=spur-cluster; SPUR_LOG_LEVEL=info; SPUR_WIPE_STATE=false
ACCT_DB_NAME=spur; ACCT_DB_USER=spur; ACCT_DB_PASSWORD=spur
ROLLING_BATCH_SIZE=1                                  # Step 12 only — agents upgraded per batch

HOSTS_ALL=( $(printf '%s\n' "${CONTROLLERS[@]}" "${AGENTS[@]}" "${LOGIN[@]}" | sort -u) )
ha_enabled=false; [ ${#CONTROLLERS[@]} -gt 1 ] && ha_enabled=true

Step 1: preflight all hosts

Run on every unique host. Abort the whole deploy on any failure.

for tgt in "${HOSTS_ALL[@]}"; do
  echo "############ $tgt ############"
  ssh -o ConnectTimeout=10 "$tgt" '
    set +e
    echo "host=$(hostname -s) fqdn=$(hostname -f)"
    echo "kernel=$(uname -r) nproc=$(nproc)"
    echo "--- spur ports (6817/6818/6821) ---"
    ss -tlnpH 2>/dev/null | grep -E ":(6817|6818|6821)\b" || echo "spur ports free"
    echo "--- existing spur pids ---"
    pgrep -ax spurctld; pgrep -ax spurd; pgrep -ax spurdbd; echo "(end pids)"
    echo "--- tools ---"
    for t in curl tar bash ss pgrep pkill systemctl; do command -v $t >/dev/null || echo "MISSING:$t"; done
    echo "--- sudo ---"; sudo -n true 2>/dev/null && echo "sudo:ok" || echo "sudo:NEEDS-PASSWORD"
    echo "--- ip ---"
    ip -4 -o addr show | awk "{print \$2, \$4}" | grep -v "127.0.0.1"
    echo "--- os ---"
    . /etc/os-release 2>/dev/null && echo "$PRETTY_NAME"
  '
done

Fail-fast rules:

• A spur port held by a process that is NOT spurctld/spurd/spurdbd → abort.
• MISSING:systemctl → abort (this skill installs systemd units; systemd is required).
• MISSING:curl/tar → abort unless SPUR_BINARY_SRC is set (installer needs them; the binary-copy path does not).
• SSH fails → abort that host; fix auth first.

(Existing spur daemons are fine — Step 4 stops them.)

Step 2: install Spur binaries on all hosts (idempotent)

Two sources, same as the playbook. ROCm/spur publishes releases (https://github.com/ROCm/spur/releases) — install.sh (no SPUR_BINARY_SRC set) downloads one automatically (SPUR_VERSION=latest by default, or nightly for a mainline build, or a specific vX.Y.Z). Set SPUR_BINARY_SRC to a local dir holding pre-built spur, spurctld, spurd, and optionally spur_mpi_pmix.so, instead when you need mainline changes not yet released, an air-gapped install, or a custom build.

resolve_mpi_plugin_src() {
  local d="$1"
  for p in \
    "$d/spur_mpi_pmix.so" \
    "$d/libspur_mpi_pmix.so" \
    "$d/lib/spur/spur_mpi_pmix.so" \
    "$(dirname "$d")/lib/spur/spur_mpi_pmix.so"; do
    [ -f "$p" ] && { echo "$p"; return 0; }
  done
  return 1
}

install_mpi_plugin_on_agent() {
  local tgt="$1" src="${2:-}"
  ssh "$tgt" "sudo mkdir -p ${SPUR_MPI_PLUGIN_DIR}"
  if [ -n "$src" ]; then
    scp -q "$src" "${tgt}:/tmp/spur_mpi_pmix.so.spur-push"
    ssh "$tgt" "sudo install -m 0755 /tmp/spur_mpi_pmix.so.spur-push ${SPUR_MPI_PLUGIN_DIR}/spur_mpi_pmix.so && rm -f /tmp/spur_mpi_pmix.so.spur-push"
    return 0
  fi
  local prefix="${SPUR_INSTALL_DIR%/*}"
  ssh "$tgt" "
    set -euo pipefail
    if [ -f ${prefix}/lib/spur/spur_mpi_pmix.so ]; then
      sudo install -m 0755 ${prefix}/lib/spur/spur_mpi_pmix.so ${SPUR_MPI_PLUGIN_DIR}/spur_mpi_pmix.so
    fi
  "
}

for tgt in "${HOSTS_ALL[@]}"; do
  ssh "$tgt" "
    set -euo pipefail
    sudo mkdir -p ${SPUR_HOME} ${SPUR_HOME}/state ${SPUR_HOME}/log ${SPUR_HOME}/etc ${SPUR_INSTALL_DIR}
  "
  if [ -n "$SPUR_BINARY_SRC" ]; then
    # Push pre-built binaries from the operator box. scp can't land a file directly under
    # /root (it runs as the plain SSH user), so stage in /tmp and sudo-install from there.
    for b in spur spurctld spurd; do
      scp -q "${SPUR_BINARY_SRC}/${b}" "${tgt}:/tmp/${b}.spur-push"
      ssh "$tgt" "sudo install -m 0755 /tmp/${b}.spur-push ${SPUR_INSTALL_DIR}/${b} && rm -f /tmp/${b}.spur-push"
    done
  else
    ssh "$tgt" "
      set -euo pipefail
      if ! sudo test -x ${SPUR_INSTALL_DIR}/spur; then
        curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh \
          | sudo INSTALL_DIR=${SPUR_INSTALL_DIR} bash -s -- ${SPUR_VERSION}
      fi
    "
  fi
  # Verify + create Slurm-compatible symlinks (the single `spur` binary dispatches on argv[0]).
  ssh "$tgt" "
    set -euo pipefail
    sudo test -x ${SPUR_INSTALL_DIR}/spur || { echo 'spur binary missing after install' >&2; exit 1; }
    for n in sbatch squeue sinfo scancel sacct sacctmgr scontrol salloc srun sattach scrontab sdiag smd sprio sreport sshare sstat strigger; do
      sudo ln -sf ${SPUR_INSTALL_DIR}/spur ${SPUR_INSTALL_DIR}/\$n
    done
    echo 'spur installed + symlinks created'
  "
done

# MPI PMIx plugin — agents only, at SPUR_MPI_PLUGIN_DIR (default /usr/lib/spur).
for tgt in "${AGENTS[@]}"; do
  if [ -n "$SPUR_BINARY_SRC" ]; then
    src=$(resolve_mpi_plugin_src "$SPUR_BINARY_SRC" || true)
    if [ -n "$src" ]; then
      install_mpi_plugin_on_agent "$tgt" "$src"
    else
      echo "note: no spur_mpi_pmix.so in SPUR_BINARY_SRC — skipping MPI plugin on $tgt"
    fi
  else
    install_mpi_plugin_on_agent "$tgt" ""
  fi
done

Do NOT rely on spur --version — it is not a supported flag and errors. Check for the file with test -x instead.