Imported from ROCm/spur-toolkit (skills/deploy-spur/SKILL.md). Install upstream with npx skills add ROCm/spur-toolkit --skill deploy-spur. Copyright stays with the author.

Deploy Spur Cluster

Spur is an AI-native job scheduler with these daemons:

• spurctld — controller / scheduler / Raft consensus (1 instance, or ≥ 3 for HA). Also serves accounting (sacct/fairshare, backed by PostgreSQL) in-process on its own gRPC port whenever [accounting].database_url is set — there is no separate accounting daemon. Only Postgres itself is a distinct service, on ACCT_HOST (default: first controller; may be a dedicated node).
• spurd — node agent, runs on every compute host
• spurstepd — per-(job, step) supervisor that spurd spawns, on every compute host. It owns the job's process tree, cgroup and exit status, and outlives the agent so a restart or upgrade of spurd doesn't kill running work. It must be installed in the same directory as spurd: the agent resolves it beside its own executable and does not search $PATH, so a node without it fails every job launch. Builds before it existed ship no such binary; that's fine, and this skill installs it only when the source provides one.

Pre-merge Spur builds additionally shipped a standalone spurdbd accounting daemon; upstream folded it into spurctld. If you're upgrading a cluster that still has spurdbd.service active, see Step 5b: migrating off a standalone spurdbd below — don't skip it, or you'll end up with two accounting paths fighting over the same Postgres.

This skill stands a cluster up with only SSH + bash on the targets. It is the standalone equivalent of ansible/ and reaches the same end state: daemons run as systemd services (survive reboot), Slurm-compatible CLI names are symlinked, and accounting is optional (default on). One flow covers all four topologies — only the host list, Raft topology, and which hosts run an agent differ. A rolling-upgrade flow (Step 12) upgrades an already-running cluster one host at a time instead of bouncing everything at once.

Defaults (override only if the user asks):

Every command below is written to work whether the SSH user is root or a non-root user with (passwordless) sudo — confirm sudo -n true succeeds during Step 1's preflight. SPUR_HOME/SPUR_INSTALL_DIR default under /root, which is mode 0700: a non-root user cannot even cd/execute/test -x into it, let alone write there, and no chown of a subdirectory fixes this (the block is on traversing /root itself). So:

• Every remote command that reads/writes under /root, /etc/systemd/system, or runs spur/sbatch/sacct/etc. must be sudo-prefixed — not just the file-writing steps. This applies uniformly (as root, sudo is a harmless no-op).
• scp and heredoc redirects (cat > /path <<EOF) run as the plain SSH user and cannot land a file directly under /root even with the SSH user later sudo-reading it. Copy to /tmp first, then sudo install/sudo mv it into place.
• Alternatively, set SPUR_INSTALL_DIR/SPUR_HOME to a world-traversable path (e.g. /opt/spur) up front to sidestep all of this — but then every install/sudo note below is still harmless, just unnecessary.

Password-based SSH works too — every ssh/scp command in this skill is a plain invocation with no auth-method assumptions baked in, so if key-based auth isn't set up, prefix each one with sshpass -p "$SSH_PASSWORD" ssh -o StrictHostKeyChecking=no ... (and the scp equivalent). Do not add -o BatchMode=yes anywhere — it disables SSH's password prompt outright and silently breaks password auth even with sshpass supplying the answer.

Step 0: gather inputs (MANDATORY — do not skip)

Before any SSH, ask the user (use AskUserQuestion for anything they didn't state; don't guess):

1. Deployment mode — pick exactly one:

   Mode	Use when
   single-node	one host runs controller and agent
   multi-node	1 controller, N compute agents (controller may also run an agent — hyperconverged)
   ha	≥ 3 controllers (Raft), N agents; controllers may be hyperconverged or dedicated (separate compute)

2. Hosts — for each role:

   • CONTROLLERS — SSH targets running spurctld. Counts: single-node 1, multi-node 1, ha odd ≥ 3.
   • AGENTS — SSH targets running spurd. Any number ≥ 1. A host may appear in both lists (hyperconverged) or only in AGENTS (dedicated compute / "separate compute" HA).

3. Accounting — deploy PostgreSQL for sacct/fairshare (served by spurctld itself, no separate daemon)? Default yes. If no, set ACCOUNTING=false; job submission still works, only sacct is unavailable.

4. Transport — direct (LAN, default) or wireguard (encrypted mesh). WireGuard adds Step 2b; everything else is identical (config advertises WG IPs instead of LAN IPs).

For HA, warn the user if controller count is even or < 3:

• N=1 → not HA; suggest multi-node.
• N=2 → "zero fault tolerance" (quorum 2, tolerates 0 failures) — code-path testing only.
• even N ≥ 4 → suggest N−1 (strictly better).

Topologies map to inventory shape exactly like the playbook:

• single-node → same host in CONTROLLERS and AGENTS
• multi-node → 1 controller, N agents
• HA hyperconverged → controllers also in AGENTS
• HA + separate compute → controllers not in AGENTS; distinct agent hosts

Once gathered, define the arrays the rest of the skill uses:

CONTROLLERS=( user@host1 user@host2 user@host3 )   # ordered — index = Raft node_id - 1; ORDER MUST BE STABLE
AGENTS=(     user@host4 )                            # may overlap CONTROLLERS (hyperconverged) or be disjoint
LOGIN=(      )                                        # OPTIONAL dedicated submission/login nodes: CLI only, no daemon. Empty = none.
SSH_USER=root
TRANSPORT=direct                                     # or wireguard
ACCOUNTING=true                                      # or false
ACCT_HOST="${CONTROLLERS[0]}"                         # accounting host: default first controller; may be ANY host — a controller, an agent, or a dedicated node (add it to HOSTS_ALL if dedicated)

SPUR_HOME=/root/spur
SPUR_INSTALL_DIR=/root/.local/bin
SPUR_VERSION=latest
SPUR_BINARY_SRC=                                     # e.g. /tmp/spur-bin to push pre-built binaries
SPUR_MPI_PLUGIN_DIR=/usr/lib/spur
SPUR_CONTROLLER_PORT=6817; SPUR_AGENT_PORT=6818; SPUR_RAFT_PORT=6821; ACCT_DB_PORT=5432
SPUR_CLUSTER_NAME=spur-cluster; SPUR_LOG_LEVEL=info; SPUR_WIPE_STATE=false
ACCT_DB_NAME=spur; ACCT_DB_USER=spur; ACCT_DB_PASSWORD=spur
ROLLING_BATCH_SIZE=1                                  # Step 12 only — agents upgraded per batch

HOSTS_ALL=( $(printf '%s\n' "${CONTROLLERS[@]}" "${AGENTS[@]}" "${LOGIN[@]}" | sort -u) )
ha_enabled=false; [ ${#CONTROLLERS[@]} -gt 1 ] && ha_enabled=true

Step 1: preflight all hosts

Run on every unique host. Abort the whole deploy on any failure.

for tgt in "${HOSTS_ALL[@]}"; do
  echo "############ $tgt ############"
  ssh -o ConnectTimeout=10 "$tgt" '
    set +e
    echo "host=$(hostname -s) fqdn=$(hostname -f)"
    echo "kernel=$(uname -r) nproc=$(nproc)"
    echo "--- spur ports (6817/6818/6821) ---"
    ss -tlnpH 2>/dev/null | grep -E ":(6817|6818|6821)\b" || echo "spur ports free"
    echo "--- existing spur pids ---"
    pgrep -ax spurctld; pgrep -ax spurd; pgrep -ax spurdbd; echo "(end pids)"
    echo "--- tools ---"
    for t in curl tar bash ss pgrep pkill systemctl; do command -v $t >/dev/null || echo "MISSING:$t"; done
    echo "--- sudo ---"; sudo -n true 2>/dev/null && echo "sudo:ok" || echo "sudo:NEEDS-PASSWORD"
    echo "--- ip ---"
    ip -4 -o addr show | awk "{print \$2, \$4}" | grep -v "127.0.0.1"
    echo "--- os ---"
    . /etc/os-release 2>/dev/null && echo "$PRETTY_NAME"
  '
done

Fail-fast rules:

• A spur port held by a process that is NOT spurctld/spurd/spurdbd → abort.
• MISSING:systemctl → abort (this skill installs systemd units; systemd is required).
• MISSING:curl/tar → abort unless SPUR_BINARY_SRC is set (installer needs them; the binary-copy path does not).
• SSH fails → abort that host; fix auth first.

(Existing spur daemons are fine — Step 4 stops them.)

Step 2: install Spur binaries on all hosts (idempotent)

Two sources, same as the playbook. ROCm/spur publishes releases (https://github.com/ROCm/spur/releases) — install.sh (no SPUR_BINARY_SRC set) downloads one automatically (SPUR_VERSION=latest by default, or nightly for a mainline build, or a specific vX.Y.Z). Set SPUR_BINARY_SRC to a local dir holding pre-built spur, spurctld, spurd, spurstepd, and optionally spur_mpi_pmix.so, instead when you need mainline changes not yet released, an air-gapped install, or a custom build. Build it with cargo build --release -p spur-cli -p spurctld -p spurd -p spur-stepd — omitting -p spur-stepd produces a spurd whose every job launch then fails on the target.

resolve_mpi_plugin_src() {
  local d="$1"
  for p in \
    "$d/spur_mpi_pmix.so" \
    "$d/libspur_mpi_pmix.so" \
    "$d/lib/spur/spur_mpi_pmix.so" \
    "$(dirname "$d")/lib/spur/spur_mpi_pmix.so"; do
    [ -f "$p" ] && { echo "$p"; return 0; }
  done
  return 1
}

install_mpi_plugin_on_agent() {
  local tgt="$1" src="${2:-}"
  ssh "$tgt" "sudo mkdir -p ${SPUR_MPI_PLUGIN_DIR}"
  if [ -n "$src" ]; then
    scp -q "$src" "${tgt}:/tmp/spur_mpi_pmix.so.spur-push"
    ssh "$tgt" "sudo install -m 0755 /tmp/spur_mpi_pmix.so.spur-push ${SPUR_MPI_PLUGIN_DIR}/spur_mpi_pmix.so && rm -f /tmp/spur_mpi_pmix.so.spur-push"
    return 0
  fi
  local prefix="${SPUR_INSTALL_DIR%/*}"
  ssh "$tgt" "
    set -euo pipefail
    if [ -f ${prefix}/lib/spur/spur_mpi_pmix.so ]; then
      sudo install -m 0755 ${prefix}/lib/spur/spur_mpi_pmix.so ${SPUR_MPI_PLUGIN_DIR}/spur_mpi_pmix.so
    fi
  "
}

for tgt in "${HOSTS_ALL[@]}"; do
  ssh "$tgt" "
    set -euo pipefail
    sudo mkdir -p ${SPUR_HOME} ${SPUR_HOME}/state ${SPUR_HOME}/log ${SPUR_HOME}/etc ${SPUR_INSTALL_DIR}
  "
  if [ -n "$SPUR_BINARY_SRC" ]; then
    # Push pre-built binaries from the operator box. scp can't land a file directly under
    # /root (it runs as the plain SSH user), so stage in /tmp and sudo-install from there.
    # spurstepd only when the build has one — a pinned older release ships none.
    # It goes to the same directory as spurd, in the same loop, so the pair on a
    # node always comes from one build: a supervisor written by one build is not
    # adopted by another.
    push_bins=(spur spurctld spurd)
    [ -f "${SPUR_BINARY_SRC}/spurstepd" ] && push_bins+=(spurstepd)
    for b in "${push_bins[@]}"; do
      scp -q "${SPUR_BINARY_SRC}/${b}" "${tgt}:/tmp/${b}.spur-push"
      ssh "$tgt" "sudo install -m 0755 /tmp/${b}.spur-push ${SPUR_INSTALL_DIR}/${b} && rm -f /tmp/${b}.spur-push"
    done
  else
    ssh "$tgt" "
      set -euo pipefail
      if ! sudo test -x ${SPUR_INSTALL_DIR}/spur; then
        curl -fsSL https://raw.githubusercontent.com/ROCm/spur/main/install.sh \
          | sudo INSTALL_DIR=${SPUR_INSTALL_DIR} bash -s -- ${SPUR_VERSION}
      fi
    "
  fi
  # Verify + create Slurm-compatible symlinks (the single `spur` binary dispatches on argv[0]).
  ssh "$tgt" "
    set -euo pipefail
    sudo test -x ${SPUR_INSTALL_DIR}/spur || { echo 'spur binary missing after install' >&2; exit 1; }
    for n in sbatch squeue sinfo scancel sacct sacctmgr scontrol salloc srun sattach scrontab sdiag smd sprio sreport sshare sstat strigger; do
      sudo ln -sf ${SPUR_INSTALL_DIR}/spur ${SPUR_INSTALL_DIR}/\$n
    done
    echo 'spur installed + symlinks created'
  "
done

# MPI PMIx plugin — agents only, at SPUR_MPI_PLUGIN_DIR (default /usr/lib/spur).
for tgt in "${AGENTS[@]}"; do
  if [ -n "$SPUR_BINARY_SRC" ]; then
    src=$(resolve_mpi_plugin_src "$SPUR_BINARY_SRC" || true)
    if [ -n "$src" ]; then
      install_mpi_plugin_on_agent "$tgt" "$src"
    else
      echo "note: no spur_mpi_pmix.so in SPUR_BINARY_SRC — skipping MPI plugin on $tgt"
    fi
  else
    install_mpi_plugin_on_agent "$tgt" ""
  fi
done

Do NOT rely on spur --version — it is not a supported flag and errors. Check for the file with test -x instead.

spurstepd check. After installing, confirm every agent has it beside spurd: ssh "$ag" "sudo test -x ${SPUR_INSTALL_DIR}/spurstepd". If it's absent, say so plainly rather than continuing quietly: on a build that expects one, every job launch on that node fails. On a build that predates it, absence is correct and nothing is wrong.

MPI plugin: release/nightly tarballs ship lib/spur/spur_mpi_pmix.so. install.sh places it under $(dirname SPUR_INSTALL_DIR)/lib/spur/; this step copies it to ${SPUR_MPI_PLUGIN_DIR} where spurd looks by default. --mpi=pmix jobs still need libpmix and Open MPI on the agent — not installed here.

Optional: prepend ${SPUR_INSTALL_DIR} to /etc/environment so non-interactive SSH gets spur/sbatch/etc. on PATH.

Step 2b: WireGuard mesh (only when TRANSPORT=wireguard)

Skip entirely for direct. WireGuard uses the built-in spur net CLI. These steps cover the single-controller (hub-and-spoke) case only. Multi-controller HA over WireGuard is possible — spur net mesh programs a full node↔node mesh so Raft works across controllers — but it's not wired into this standalone skill; use the Ansible spur_wireguard role (see ansible/README.md) for HA-over-mesh. Steps here, using the real CLI (all spur net commands log to stderr):

1. apt install wireguard-tools on every host.
2. On the controller: spur net init --cidr 10.44.0.0/16 --port 51820 --interface spur0 (auto-assigns .1). Read its pubkey with wg show spur0 public-key (there is no spur net pubkey).
3. On each agent (assign .2, .3, …): spur net join --endpoint <ctl-ip>:51820 --server-key <ctl-pubkey> --address 10.44.0.<N> --prefix-len 16 --interface spur0. --prefix-len must match the CIDR (defaults to 16). Read the agent pubkey with wg show spur0 public-key.
4. On the controller, register each agent: spur net add-peer --key <agent-pubkey> --allowed-ip 10.44.0.<N>/32 --interface spur0.

Then set WG_IP[$host] per host and use those in place of IP[...] for [controller].hosts, peers, and spurd --address/--controller. There is no spur net down — tear down with wg-quick down spur0 (or ip link del spur0) and remove /etc/wireguard/spur0.conf. If the user wants WG but you cannot verify mesh connectivity (all hosts on one /24 makes it moot), tell them and offer direct instead.

Step 3: derive per-host facts (hostnames, IPs, node_ids)

host_short() { ssh "$1" 'hostname -s'; }
host_addr()  { local t="${1#*@}"; echo "$t"; }   # SSH target IP/host, minus user@

declare -A SHORT IP NODE_ID
for h in "${HOSTS_ALL[@]}"; do
  SHORT[$h]=$(host_short "$h")
  IP[$h]=$(host_addr "$h")          # for TRANSPORT=wireguard, set IP[$h]=${WG_IP[$h]} instead
done

# 1-based Raft node_id = position in CONTROLLERS. ORDER MATTERS — reordering after a
# deploy breaks openraft membership. To re-order, wipe state on every controller and redeploy.
for i in "${!CONTROLLERS[@]}"; do NODE_ID[${CONTROLLERS[$i]}]=$((i+1)); done

hostname -s (not -f) is intentional — the controller's [[nodes]] names, spurd --hostname, and spur show node <name> must all use the same short form.

Step 4: stop existing daemons + wipe state

Stop via systemd if a unit exists, and belt-and-suspenders pkill -x (exact name — pkill -f spurd also kills spurctld).

for tgt in "${HOSTS_ALL[@]}"; do
  ssh "$tgt" '
    # spurstepd is NOT in this list: a supervisor is meant to outlive its agent,
    # and a stop here is followed by a restart. Kill them only where the host is
    # actually going away — see the teardown note at the end of this skill.
    for svc in spurd spurctld spurdbd; do
      sudo systemctl stop "$svc" 2>/dev/null || true
    done
    sudo pkill -x spurd 2>/dev/null || true
    sudo pkill -x spurctld 2>/dev/null || true
    sudo pkill -x spurdbd 2>/dev/null || true
    for i in $(seq 1 10); do
      pgrep -x spurctld >/dev/null || pgrep -x spurd >/dev/null || pgrep -x spurdbd >/dev/null || exit 0
      sleep 0.5
    done
    echo "daemons still running after 5s" >&2; exit 1
  '
done

# Wipe Raft state on controllers BEFORE start (so spurctld does not rewrite the log we delete).
if [ "$SPUR_WIPE_STATE" = true ]; then
  for tgt in "${CONTROLLERS[@]}"; do
    ssh "$tgt" "sudo rm -rf ${SPUR_HOME}/state && sudo mkdir -p ${SPUR_HOME}/state"
  done
fi

Default is no wipe so re-runs and upgrades preserve the job queue and node registrations. Wipe only for a fresh install or an intentional Raft reinit. Because Spur 0.3.0 has no online Raft membership change, changing the controller set (add/remove/reorder) requires a wipe — if you're keeping state but the controller list differs from the running cluster, warn the user and require SPUR_WIPE_STATE=true. Compute agents are not Raft members and can be added/removed freely without a wipe. When demoting a host from controller to agent-only, systemctl disable --now spurctld on it first, or the stale daemon keeps the old membership and can block quorum.

Step 5: deploy accounting (only when ACCOUNTING=true) — on ACCT_HOST

Accounting is served in-process by every controller's spurctld (no separate daemon); only Postgres is a distinct service, and it lives on ACCT_HOST (default CONTROLLERS[0], but may be any host — a controller, an agent, or a dedicated node). Deploy it before the controllers so Postgres is reachable when spurctld's embedded accounting service connects. Idempotent: existence-checked role/DB creation.

Because every controller — not just ACCT_HOST — connects to this Postgres directly over the network, it must accept remote TCP connections, not just localhost:

# pg_hba lines granting each controller's IP access to the spur DB — built locally,
# same pattern as the [[nodes]] blocks / CSVs in Step 6. Each is its own dedup-append
# statement (grep -qxF before appending) so re-running Step 5 doesn't pile up duplicate
# lines the way a bare `tee -a` would — mirrors Ansible's lineinfile exact-match semantics.
#
# pg_hba.conf's address field is a literal IP/CIDR, not a hostname, so this can't just
# reuse IP[$h] — CONTROLLERS[] entries are SSH targets and may legitimately be DNS
# names (host_addr() in Step 3 doesn't resolve them, it only strips user@). Appending
# "somehost.example.com/32" produces an unparseable pg_hba line and takes Postgres
# down on the next restart. Ask each controller for its own real IP instead — this is
# also the more correct choice regardless of hostnames: it's the address Postgres will
# actually see as the connection's source, which may differ from whatever address SSH
# uses to reach the host (a management VLAN, a NAT'd address, etc.).
pg_hba_appends=""
for h in "${CONTROLLERS[@]}"; do
  ctl_ip=$(ssh "$h" "hostname -I | awk '{print \$1}'")
  line="host    ${ACCT_DB_NAME}    ${ACCT_DB_USER}    ${ctl_ip}/32    scram-sha-256"
  pg_hba_appends+="sudo grep -qxF '${line}' \"\$pg_hba\" || echo '${line}' | sudo tee -a \"\$pg_hba\" >/dev/null"$'\n'
done

if [ "$ACCOUNTING" = true ]; then
  ssh "$ACCT_HOST" "
    set -euo pipefail
    export DEBIAN_FRONTEND=noninteractive
    # Install PostgreSQL (Debian/Ubuntu). For RHEL, swap in dnf + postgresql-server + initdb.
    if ! command -v psql >/dev/null 2>&1; then
      sudo apt-get update -qq
      sudo apt-get install -y -qq postgresql postgresql-contrib
    fi
    sudo systemctl enable --now postgresql
    # Create role + DB idempotently via the postgres superuser.
    sudo -u postgres psql -tAc \"SELECT 1 FROM pg_roles WHERE rolname='${ACCT_DB_USER}'\" | grep -q 1 \
      || sudo -u postgres psql -c \"CREATE ROLE ${ACCT_DB_USER} LOGIN PASSWORD '${ACCT_DB_PASSWORD}'\"
    sudo -u postgres psql -tAc \"SELECT 1 FROM pg_database WHERE datname='${ACCT_DB_NAME}'\" | grep -q 1 \
      || sudo -u postgres psql -c \"CREATE DATABASE ${ACCT_DB_NAME} OWNER ${ACCT_DB_USER}\"

    # Listen on all interfaces and allow every controller's IP to authenticate.
    # /etc/postgresql/<version>/main/postgresql.conf is 3 levels below /etc/postgresql.
    pg_conf=\$(sudo find /etc/postgresql -maxdepth 3 -name postgresql.conf | head -1)
    pg_hba=\$(dirname \"\$pg_conf\")/pg_hba.conf
    sudo sed -i \"s/^#\\?\\s*listen_addresses\\s*=.*/listen_addresses = '*'/\" \"\$pg_conf\"
    grep -q listen_addresses \"\$pg_conf\" || echo \"listen_addresses = '*'\" | sudo tee -a \"\$pg_conf\" >/dev/null
    ${pg_hba_appends}
    sudo systemctl restart postgresql
    for i in \$(seq 1 30); do ss -tlnH | grep -q ':${ACCT_DB_PORT}\b' && { echo 'postgres up'; break; }; sleep 1; [ \$i -eq 30 ] && { echo 'postgres did not bind ${ACCT_DB_PORT}' >&2; exit 1; }; done
  "
fi

Step 5b: migrating off a standalone spurdbd (pre-merge upgrades only)

Skip this if the cluster has never run the old, separate spurdbd daemon. If it has (check systemctl is-active spurdbd on ACCT_HOST, or an [accounting] host = ... line in spur.conf), clean it up on ACCT_HOST — otherwise the old daemon keeps running (harmlessly, but confusingly) alongside the new embedded accounting:

ssh "$ACCT_HOST" "
  sudo systemctl disable --now spurdbd 2>/dev/null || true
  sudo rm -f /etc/systemd/system/spurdbd.service
  sudo systemctl daemon-reload
  sudo rm -f ${SPUR_INSTALL_DIR}/spurdbd
"

The old pre-merge spur_binary_src push (Step 2) copied spurdbd to every host in HOSTS_ALL, not just ACCT_HOST — it only ever ran as a service on ACCT_HOST, but a stray unused binary can be left on other controllers/agents too. Sweep it everywhere while you're at it:

for tgt in "${HOSTS_ALL[@]}"; do
  ssh "$tgt" "sudo rm -f ${SPUR_INSTALL_DIR}/spurdbd"
done

Run this before (or as part of) Step 5, then proceed to Step 6 — the regenerated spur.conf (below) already drops the old host = ... key.

Step 6: render spur.conf + install spurctld systemd unit on every controller

All controllers get the same spur.conf except node_id (HA only). The peers list order must be identical on every host (by node_id).

# CSVs for the controller list and the raft peers list.
ctl_hosts_csv=""; peers_csv=""
for h in "${CONTROLLERS[@]}"; do
  ctl_hosts_csv+="\"${IP[$h]}\", "
  peers_csv+="\"${IP[$h]}:${SPUR_RAFT_PORT}\", "
done
ctl_hosts_csv=${ctl_hosts_csv%, }; peers_csv=${peers_csv%, }

# Per-agent [[nodes]] blocks (90% of RAM) + partition node list.
nodes_blocks=""; part_csv=""
for h in "${AGENTS[@]}"; do
  cpus=$(ssh "$h" 'nproc')
  mem_kb=$(ssh "$h" "awk '/MemTotal/{print \$2}' /proc/meminfo")
  mem_mb=$(( mem_kb / 1024 * 9 / 10 ))
  nodes_blocks+=$'\n'"[[nodes]]"$'\n'"names = \"${SHORT[$h]}\""$'\n'"cpus = ${cpus}"$'\n'"memory_mb = ${mem_mb}"$'\n'
  part_csv+="${SHORT[$h]},"
done
part_csv=${part_csv%,}

# Accounting block (only when enabled). No separate daemon anymore — spurctld
# connects to Postgres directly, so database_url uses ACCT_HOST's network address
# (not localhost) even for a controller that happens to be ACCT_HOST itself.
# ${IP[$ACCT_HOST]} requires ACCT_HOST to be in HOSTS_ALL (Step 3); falls back to
# stripping user@ from ACCT_HOST if it wasn't resolved into the IP map.
acct_block=""
if [ "$ACCOUNTING" = true ]; then
  acct_ip="${IP[$ACCT_HOST]:-${ACCT_HOST#*@}}"
  acct_block=$'\n'"[accounting]"$'\n'"database_url = \"postgresql://${ACCT_DB_USER}:${ACCT_DB_PASSWORD}@${acct_ip}:${ACCT_DB_PORT}/${ACCT_DB_NAME}\""$'\n'"fairshare_refresh_secs = 30"$'\n'
fi

wg_line="wg_enabled = false"
[ "$TRANSPORT" = wireguard ] && wg_line="wg_enabled = true"$'\n'"wg_interface = \"spur0\""

for ctl in "${CONTROLLERS[@]}"; do
  raft_block=""
  if $ha_enabled; then
    raft_block="node_id = ${NODE_ID[$ctl]}"$'\n'"peers = [${peers_csv}]"
  fi
  tmp=$(mktemp)
  cat > "$tmp" <<EOF
cluster_name = "${SPUR_CLUSTER_NAME}"

[controller]
listen_addr = "[::]:${SPUR_CONTROLLER_PORT}"
hosts = [${ctl_hosts_csv}]
state_dir = "${SPUR_HOME}/state"
raft_listen_addr = "[::]:${SPUR_RAFT_PORT}"
${raft_block}

[scheduler]
plugin = "backfill"
interval_secs = 1
${acct_block}
[network]
${wg_line}
agent_port = ${SPUR_AGENT_PORT}
${nodes_blocks}
[[partitions]]
name = "default"
default = true
nodes = "${part_csv}"
max_time = "INFINITE"
EOF
  # scp can't land a file directly under /root — stage in /tmp and sudo-move it into place.
  scp -q "$tmp" "$ctl:/tmp/spur.conf.push"
  ssh "$ctl" "sudo mkdir -p ${SPUR_HOME}/etc && sudo mv /tmp/spur.conf.push ${SPUR_HOME}/etc/spur.conf"
  rm -f "$tmp"

  # Install + (re)start the spurctld systemd unit. Same /tmp-then-sudo-mv pattern as spur.conf —
  # a plain heredoc redirect can't write to /etc/systemd/system as a non-root SSH user.
  ssh "$ctl" "cat > /tmp/spurctld.service.tmp <<'UNIT'
[Unit]
Description=Spur Controller Daemon (spurctld)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=HOME=${SPUR_HOME}
WorkingDirectory=${SPUR_HOME}
ExecStart=${SPUR_INSTALL_DIR}/spurctld -f ${SPUR_HOME}/etc/spur.conf --state-dir ${SPUR_HOME}/state --log-level ${SPUR_LOG_LEVEL}
Restart=on-failure
RestartSec=3
User=root
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
UNIT
  sudo mv /tmp/spurctld.service.tmp /etc/systemd/system/spurctld.service
  sudo systemctl daemon-reload
  sudo systemctl enable spurctld
  sudo systemctl restart spurctld
  "
done

# Wait for every controller's gRPC port to bind.
for ctl in "${CONTROLLERS[@]}"; do
  ssh "$ctl" "for i in \$(seq 1 30); do ss -tlnH | grep -q ':${SPUR_CONTROLLER_PORT}\b' && exit 0; sleep 1; done; echo 'spurctld did not bind ${SPUR_CONTROLLER_PORT}' >&2; exit 1"
done

HA only: spurctld binds 6817 immediately but returns no leader elected yet until quorum forms. Wait on the first controller:

if $ha_enabled; then
  ssh "${CONTROLLERS[0]}" "
    for i in \$(seq 1 60); do
      out=\$(sudo ${SPUR_INSTALL_DIR}/spur nodes 2>&1); rc=\$?
      # Ready only when the command SUCCEEDS and the output has a real node table.
      # Gate on rc==0 too — a Permission denied / crash must NOT be read as 'leader up'.
      if [ \$rc -eq 0 ] && ! echo \"\$out\" | grep -qE 'no leader|not the Raft leader|cannot reach leader|transport error|Connection refused|Permission denied'; then
        echo OK; exit 0
      fi
      sleep 1
    done
    echo \"timeout waiting for leader: \$out\" >&2; exit 1
  "
fi

Set the client env on each controller so squeue/sinfo/scontrol and sacct/sacctmgr/sreport/sshare (all use SPUR_CONTROLLER_ADDR now — accounting rides the controller's own port, there's no separate accounting flag/env anymore) work with no per-command flags. List every controller, comma-separated — spur-cli rotates past a dead endpoint, so a controller can serve the CLI even when a different controller (or itself) is down. Controllers only (agents don't run the CLI for users here):

ctl_endpoints_csv=""
for c in "${CONTROLLERS[@]}"; do ctl_endpoints_csv+="http://${IP[$c]}:${SPUR_CONTROLLER_PORT},"; done
ctl_endpoints_csv=${ctl_endpoints_csv%,}

for ctl in "${CONTROLLERS[@]}"; do
  ssh "$ctl" "
    sudo sed -i '/^SPUR_CONTROLLER_ADDR=/d;/^SPUR_ACCOUNTING_ADDR=/d' /etc/environment
    echo 'SPUR_CONTROLLER_ADDR=${ctl_endpoints_csv}' | sudo tee -a /etc/environment >/dev/null
  "
done

(/etc/environment is read at login, so a fresh shell / bash -lc picks it up; for sudo also running the CLI, use sudo -E to pass the vars through. The sed also strips any stale SPUR_ACCOUNTING_ADDR left over from a pre-merge deployment of this host.)

Step 7: install spurd systemd unit on every agent

spurd --controller accepts a comma-separated endpoint list and rotates past a dead one (spur-client endpoint rotation), so every agent is pointed at every controller, not just CONTROLLERS[0] — a single surviving controller is enough (followers forward writes via Raft either way). WorkingDirectory=${SPUR_HOME} in the unit sets the fallback stdout dir for spur-<N>.out (a job's own WorkDir/submit-CWD takes precedence). --hostname/--address are explicit (auto-detect picks 127.0.0.1, breaking inter-node dispatch).

KillMode=process is required, not cosmetic: job supervisors detach from the agent into their own session but stay in the unit's cgroup, so systemd's default (control-group) kills them on every stop/restart and takes the running work with them. SPUR_STEPD_STATE_DIR is where the agent keeps the supervisor sessions it re-adopts after a restart; it is deliberately not ${SPUR_HOME}/state (the controller's Raft directory), so a hyperconverged host never has the two share one. It is set as an environment variable rather than spurd --state-dir because a build predating spurstepd ignores an unknown variable but refuses to start on an unknown flag.

CTL_ENDPOINTS="$ctl_endpoints_csv"   # reuse the list built for the client env above
for ag in "${AGENTS[@]}"; do
  # /tmp-then-sudo-mv: a plain heredoc redirect can't write to /etc/systemd/system as a
  # non-root SSH user.
  ssh "$ag" "cat > /tmp/spurd.service.tmp <<'UNIT'
[Unit]
Description=Spur Node Agent (spurd)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Environment=HOME=${SPUR_HOME}
Environment=SPUR_STEPD_STATE_DIR=${SPUR_HOME}/agent-state
WorkingDirectory=${SPUR_HOME}
ExecStart=${SPUR_INSTALL_DIR}/spurd --controller ${CTL_ENDPOINTS} --hostname ${SHORT[$ag]} --address ${IP[$ag]} --listen 0.0.0.0:${SPUR_AGENT_PORT} --log-level ${SPUR_LOG_LEVEL}
Restart=on-failure
RestartSec=3
KillMode=process
User=root
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
UNIT
  sudo mv /tmp/spurd.service.tmp /etc/systemd/system/spurd.service
  sudo systemctl daemon-reload
  sudo systemctl enable spurd
  sudo systemctl restart spurd
  for i in \$(seq 1 30); do ss -tlnH | grep -q ':${SPUR_AGENT_PORT}\b' && exit 0; sleep 1; done
  echo 'spurd did not bind ${SPUR_AGENT_PORT}' >&2; exit 1
  "
done

Step 7b: configure login (submission) nodes (only when LOGIN is non-empty)

A login node is a pure client: the spur CLI (installed in Step 2) + the controller env, no daemon. Users SSH in and run sbatch/squeue/sacct/srun. Just set SPUR_CONTROLLER_ADDR (same comma-joined controller list the controllers get in Step 6) — accounting rides that address, no separate env. Nothing else to install or start.

if [ ${#LOGIN[@]} -gt 0 ]; then
  for lg in "${LOGIN[@]}"; do
    ssh "$lg" "
      sudo sed -i '/^SPUR_CONTROLLER_ADDR=/d;/^SPUR_ACCOUNTING_ADDR=/d' /etc/environment
      echo 'SPUR_CONTROLLER_ADDR=${ctl_endpoints_csv}' | sudo tee -a /etc/environment >/dev/null
    "
  done
fi

Networking: a login node needs outbound access to the controllers on ${SPUR_CONTROLLER_PORT} (all CLI + accounting) and, for interactive srun live output, to the agents on ${SPUR_AGENT_PORT} (srun streams output directly from the agent). Under TRANSPORT=wireguard, run Step 2b's join on each login node too (assign it a mesh IP and spur net add-peer it on the controller) so this works over the tunnel.

Step 8: wait for every agent to register

spur nodes collapses by partition, so check per-agent with spur show node:

for ag in "${AGENTS[@]}"; do
  for i in $(seq 1 30); do
    if ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur show node ${SHORT[$ag]} >/dev/null 2>&1"; then
      echo "${SHORT[$ag]} registered"; break
    fi
    [ $i -eq 30 ] && { echo "${SHORT[$ag]} never registered" >&2; exit 1; }
    sleep 1
  done
done

Step 9: smoke test

Single-node job (every deploy)

ssh "${CONTROLLERS[0]}" "
sudo rm -f /tmp/spur-test-single.sh   # a same-named file from an earlier/unrelated deploy on a
                                       # shared host can be root-owned; the plain cat> below would
                                       # then fail *silently* (no set -e here) and re-run the stale
                                       # script instead of this one.
cat > /tmp/spur-test-single.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=spur-test-single
echo \"ran on \$(hostname) at \$(date)\"
EOF
chmod +x /tmp/spur-test-single.sh || { echo 'failed to write test script' >&2; exit 1; }
cd /tmp   # predictable, world-accessible WorkDir; job stdout -> /tmp/spur-<JOBID>.out. Do NOT cd into a 0700 dir like /root/spur when SSHing non-root.
jid=\$(sudo ${SPUR_INSTALL_DIR}/spur submit /tmp/spur-test-single.sh | grep -oE '[0-9]+')
echo \"JOBID=\$jid\"
for i in \$(seq 1 30); do
  st=\$(sudo ${SPUR_INSTALL_DIR}/spur show job \$jid 2>/dev/null | grep -oE 'JobState=[A-Z]+' | head -1 | cut -d= -f2)
  case \"\$st\" in COMPLETED) echo OK; break ;; FAILED|CANCELLED|TIMEOUT|NODE_FAIL) echo \"BAD: \$st\" >&2; exit 1 ;; esac
  sleep 1
done
echo \"final state: \$st\"
"

Output lands in spur-<JOBID>.out on whichever agent ran the job, in the job's WorkDir — i.e. the CWD at submit time (/tmp above). It is NOT necessarily ${SPUR_HOME}. To locate it robustly, loop agents and search the likely dirs: sudo find /tmp ${SPUR_HOME} /home /root -maxdepth 2 -name 'spur-<JOBID>.out' (no shared-FS assumption; use sudo for non-root SSH).

Multi-node job (when |AGENTS| ≥ 2)

-N must not exceed the agent count. Use <<'EOF' so $SPUR_* vars reach the agent verbatim and expand at run time; inject N via sed.

N=${#AGENTS[@]}
ssh "${CONTROLLERS[0]}" "
sudo rm -f /tmp/spur-test-multi.sh   # see single-node note on stale-file collisions
cat > /tmp/spur-test-multi.sh <<'EOF'
#!/bin/bash
#SBATCH --job-name=spur-test-multi
#SBATCH -N __N__
#SBATCH --ntasks-per-node=1
echo \"node \$SPUR_TASK_OFFSET of \$SPUR_NUM_NODES on \$(hostname); peers=\$SPUR_PEER_NODES\"
EOF
sed -i 's/__N__/${N}/' /tmp/spur-test-multi.sh
chmod +x /tmp/spur-test-multi.sh || { echo 'failed to write test script' >&2; exit 1; }
cd /tmp   # predictable WorkDir (see single-node note); output -> /tmp/spur-<JOBID>.out per node
jid=\$(sudo ${SPUR_INSTALL_DIR}/spur submit /tmp/spur-test-multi.sh | grep -oE '[0-9]+')
echo \"JOBID=\$jid\"
for i in \$(seq 1 60); do
  st=\$(sudo ${SPUR_INSTALL_DIR}/spur show job \$jid 2>/dev/null | grep -oE 'JobState=[A-Z]+' | head -1 | cut -d= -f2)
  case \"\$st\" in COMPLETED) echo OK; break ;; FAILED|CANCELLED|TIMEOUT|NODE_FAIL) echo \"BAD: \$st\" >&2; exit 1 ;; esac
  sleep 1
done
if [ \"\$st\" != COMPLETED ]; then
  echo \"multi-node job stuck in state '\$st' after 60s. Likely cause: a leftover per-job scratch\" >&2
  echo \"file (.spur_job_<id>.sh) in an agent's WorkDir from an earlier deploy/job-id reset that\" >&2
  echo \"can't be overwritten, so that agent rejects dispatch ('failed to write job script') and\" >&2
  echo \"the job sits in COMPLETING forever (node stays 'mix', not 'idle'). Fix: sudo cancel the\" >&2
  echo \"job (sudo ${SPUR_INSTALL_DIR}/spur cancel \$jid), sudo rm -f any stale .spur_job_*.sh in\" >&2
  echo \"/tmp on each agent, and resubmit.\" >&2
  exit 1
fi
"
# Multi-node writes locally on each node, in the job's WorkDir (here /tmp, since the submit
# above cd'd there) — NOT ${SPUR_HOME}. Fetch from every agent, checking both locations since
# WorkingDirectory=${SPUR_HOME} in the spurd unit is only a fallback if WorkDir is unavailable.
for ag in "${AGENTS[@]}"; do
  echo "=== ${SHORT[$ag]} ==="
  ssh "$ag" "sudo find /tmp ${SPUR_HOME} -maxdepth 2 -name 'spur-*.out' 2>/dev/null -exec sudo cat {} \;"
done

Accounting check (when ACCOUNTING=true)

ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/sacct | head"   # sacct works from any controller (accounting is served by spurctld itself)
ssh "$ACCT_HOST" "sudo -u postgres psql -d ${ACCT_DB_NAME} -tAc 'SELECT count(*) FROM jobs;'"   # postgres lives on ACCT_HOST

Expect a row per completed job. (With ACCOUNTING=false, sacct is expected to fail — that's fine; jobs still run.)

Step 10: verify

ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur nodes"
ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur queue"

# Every daemon should be systemd-active.
for ctl in "${CONTROLLERS[@]}"; do ssh "$ctl" "systemctl is-active spurctld"; done
for ag  in "${AGENTS[@]}";      do ssh "$ag"  "systemctl is-active spurd"; done
[ "$ACCOUNTING" = true ] && ssh "$ACCT_HOST" "systemctl is-active postgresql"

# HA: identify the CURRENT leader. Every node that was ever leader has a
# 'become leader' line, so grep -m1 (first match) is wrong after any
# re-election. The authoritative current leader is in each node's persisted
# vote — read node_id from vote.json (identical on all healthy peers).
if $ha_enabled; then
  ssh "${CONTROLLERS[0]}" "sudo cat ${SPUR_HOME}/state/raft/vote.json 2>/dev/null" \
    | grep -oE '\"node_id\":[0-9]+' | tail -1 | sed 's/.*://' \
    | xargs -I{} echo "current Raft leader: node_id={}"
fi

# Separate-compute HA sanity: controllers that are NOT agents must have spurd inactive.
for ctl in "${CONTROLLERS[@]}"; do
  is_agent=false; for a in "${AGENTS[@]}"; do [ "$a" = "$ctl" ] && is_agent=true; done
  $is_agent || ssh "$ctl" "systemctl is-active spurd 2>/dev/null | grep -q inactive && echo '$ctl: no agent (correct)' || echo '$ctl: unexpected spurd'"
done

Step 11: teardown (only when asked)

Job supervisors have to be killed explicitly here. A spurstepd is built to outlive its agent — its own session, reparented to init, and spared by the unit's KillMode=process — so disabling spurd leaves it running. It also ignores a bare SIGTERM by design, so the SIGKILL below is what actually lands. Kill the job's own processes first (they live in the job's cgroup, not the supervisor's): a supervisor whose workload just died reports the exit, and killing the supervisor first would throw that completion away.

for tgt in "${HOSTS_ALL[@]}"; do
  ssh "$tgt" "
    # Job processes first, then the daemons, then the supervisors they left behind.
    pids=\$(sudo find /sys/fs/cgroup/spur -name cgroup.procs 2>/dev/null | xargs -r sudo cat 2>/dev/null | sort -u)
    [ -n \"\$pids\" ] && { echo \"\$pids\" | xargs sudo kill -TERM 2>/dev/null; sleep 5; echo \"\$pids\" | xargs sudo kill -KILL 2>/dev/null; } || true
    for svc in spurd spurctld spurdbd; do sudo systemctl disable --now \$svc 2>/dev/null || true; done
    sudo pkill -x spurd 2>/dev/null; sudo pkill -x spurctld 2>/dev/null; sudo pkill -x spurdbd 2>/dev/null
    sudo pkill -x -TERM spurstepd 2>/dev/null; sleep 2; sudo pkill -x -KILL spurstepd 2>/dev/null
    sudo rm -f /etc/systemd/system/spurd.service /etc/systemd/system/spurctld.service /etc/systemd/system/spurdbd.service
    sudo systemctl daemon-reload 2>/dev/null || true
    sudo rm -rf ${SPUR_HOME}
    sudo rm -f /root/spur-*.out ${SPUR_INSTALL_DIR}/spur-*.out /tmp/spur-*.out
  "
done

To also remove accounting data (destructive): ssh "$ACCT_HOST" "sudo -u postgres dropdb ${ACCT_DB_NAME}; sudo -u postgres dropuser ${ACCT_DB_USER}". Only do this if the user explicitly asks — it deletes all job history. Leave PostgreSQL itself installed unless asked to purge it.

Step 12: rolling upgrade (only when asked to upgrade a live cluster)

Use this instead of re-running Steps 1–9 when jobs are currently running and a full-cluster daemon bounce (which Steps 1-9 do — no draining, no batching) is not acceptable. Assumes the cluster is already up and healthy; refuse to proceed otherwise. Requires SPUR_BINARY_SRC pointing at the new build (rebuild every binary together — same caveat as any upgrade; spurd and spurstepd especially, since a supervisor written by one build is not adopted by another).

spurstepd is pushed to agents alongside spurd in this step, same as any other binary — no special handling needed even for the upgrade that introduces it. Push spur_mpi_pmix.so to agents when present in SPUR_BINARY_SRC. Only exercised so far with TRANSPORT=direct; the IP[] map this step reuses from Step 3 still needs to hold real addresses (WG_IP[] per Step 2b) for a wireguard cluster — re-derive it in this shell session first if it isn't already populated.

This step only pushes new binaries and restarts daemons — it does not touch spur.conf or Postgres. If the cluster is still on the pre-merge standalone-spurdbd architecture, do the Step 5b migration first (a full-flow, bounce-based operation: Steps 2/4/5b/5/6/7/8/9/10) and confirm it's healthy on the merged-accounting build before using this step for further low-disruption upgrades. Running this step against a still-unmigrated cluster would push a spurctld binary that expects the new embedded-accounting config shape without updating spur.conf/pg_hba.conf to match — don't do that.

# Guard rail: refuse a rolling upgrade if state would be wiped, or the cluster
# isn't already healthy.
[ "$SPUR_WIPE_STATE" = true ] && { echo "rolling upgrade must not wipe Raft state" >&2; exit 1; }
ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur nodes >/dev/null" \
  || { echo "cluster is not healthy before starting — investigate first" >&2; exit 1; }

# --- Controllers, ONE AT A TIME (never in parallel — that would drop Raft quorum) ---
for ctl in "${CONTROLLERS[@]}"; do
  echo "=== upgrading controller ${SHORT[$ctl]} ==="
  for b in spur spurctld spurd; do
    scp -q "${SPUR_BINARY_SRC}/${b}" "${ctl}:/tmp/${b}.spur-push"
    ssh "$ctl" "sudo install -m 0755 /tmp/${b}.spur-push ${SPUR_INSTALL_DIR}/${b} && rm -f /tmp/${b}.spur-push"
  done
  ssh "$ctl" "sudo systemctl restart spurctld"
  ssh "$ctl" "for i in \$(seq 1 30); do ss -tlnH | grep -q ':${SPUR_CONTROLLER_PORT}\b' && exit 0; sleep 1; done; exit 1" \
    || { echo "${SHORT[$ctl]} did not come back up — aborting rolling upgrade" >&2; exit 1; }
  # Health gate before moving to the next controller: leader must be elected again.
  # Client-side failover means the OTHER controllers keep serving agents/CLI while
  # this one is down, so this is the only wait needed between controllers.
  if $ha_enabled; then
    ssh "${CONTROLLERS[0]}" "
      for i in \$(seq 1 60); do
        out=\$(sudo ${SPUR_INSTALL_DIR}/spur nodes 2>&1); rc=\$?
        [ \$rc -eq 0 ] && ! echo \"\$out\" | grep -qE 'no leader|not the Raft leader|cannot reach leader|transport error|Connection refused' && { echo OK; exit 0; }
        sleep 1
      done
      echo 'timeout waiting for leader after controller restart' >&2; exit 1
    " || exit 1
  fi
done

# --- Agents, in batches of ROLLING_BATCH_SIZE (default 1) ---
i=0
while [ $i -lt ${#AGENTS[@]} ]; do
  batch=("${AGENTS[@]:i:ROLLING_BATCH_SIZE}")
  echo "=== draining batch: ${batch[*]} ==="
  for ag in "${batch[@]}"; do
    ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur node drain ${SHORT[$ag]} --reason 'rolling upgrade'"
  done
  for ag in "${batch[@]}"; do
    for j in $(seq 1 120); do
      st=$(ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur show node ${SHORT[$ag]} 2>/dev/null" \
           | awk -v n="${SHORT[$ag]}" '/^NodeName=/{p=($0=="NodeName="n)} p' | grep -oE 'State=[A-Z]+' | head -1 | cut -d= -f2)
      [ "$st" = "DRAINED" ] && break
      [ $j -eq 120 ] && { echo "${SHORT[$ag]} did not drain (still running jobs after 120s)" >&2; exit 1; }
      sleep 1
    done
  done
  for ag in "${batch[@]}"; do
    push_bins=(spur spurctld spurd)
    [ -f "${SPUR_BINARY_SRC}/spurstepd" ] && push_bins+=(spurstepd)
    for b in "${push_bins[@]}"; do
      scp -q "${SPUR_BINARY_SRC}/${b}" "${ag}:/tmp/${b}.spur-push"
      ssh "$ag" "sudo install -m 0755 /tmp/${b}.spur-push ${SPUR_INSTALL_DIR}/${b} && rm -f /tmp/${b}.spur-push"
    done
    src=$(resolve_mpi_plugin_src "$SPUR_BINARY_SRC" || true)
    if [ -n "$src" ]; then
      install_mpi_plugin_on_agent "$ag" "$src"
    fi
    ssh "$ag" "sudo systemctl restart spurd"
  done
  for ag in "${batch[@]}"; do
    for j in $(seq 1 30); do
      ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/spur show node ${SHORT[$ag]} >/dev/null 2>&1" && break
      [ $j -eq 30 ] && { echo "${SHORT[$ag]} never re-registered after upgrade" >&2; exit 1; }
      sleep 1
    done
    ssh "${CONTROLLERS[0]}" "sudo ${SPUR_INSTALL_DIR}/scontrol update NodeName=${SHORT[$ag]} State=RESUME"
  done
  i=$((i + ROLLING_BATCH_SIZE))
done

# Confirm the upgraded cluster actually schedules work (reuse Step 9's single-node test).

Notes:

• Drain, don't kill. spur node drain stops new scheduling onto a node and lets its running jobs finish on their own (DRAINING → DRAINED); it does not evict them. Only touch the binary/restart spurd once a node reaches DRAINED.
• A drained node stays drained after spurd restarts — draining is server-side state, not tied to the agent process. The scontrol update ... State=RESUME step is mandatory, or upgraded capacity sits idle indefinitely.
• Controllers never restart in parallel. With < 3 controllers there's no other controller to fail over to during the restart, so a rolling upgrade of a single-controller (or 2-controller) cluster is still a short outage — this only buys zero-downtime with a real HA quorum (≥ 3).
• If any step aborts partway, the cluster is left in a safe, valid state (some hosts upgraded, some not, nothing drained-and-forgotten except mid-batch — check spur nodes for any node still DRAIN/DRAINING and RESUME it manually before retrying).

Gotchas (all hard-won — don't relearn them)

Install / systemd

• This skill uses systemd, not nohup. Units are /etc/systemd/system/spur{ctld,d}.service (spurdbd.service only exists as a leftover from a pre-merge deployment — see Step 5b), enabled (survive reboot), Restart=on-failure. Always systemctl daemon-reload after writing a unit.
• spur --version is NOT supported — it errors. Check the binary with test -x, not by running --version.
• install.sh now works (ROCm/spur publishes releases) — leaving SPUR_BINARY_SRC unset downloads one automatically. Set SPUR_BINARY_SRC to a local dir of pre-built binaries instead for mainline changes not yet released, air-gapped installs, or a custom build.
• Slurm-compat symlinks — all 18 names the spur multi-call binary recognizes via argv[0] dispatch (sbatch, squeue, sinfo, scancel, sacct, sacctmgr, scontrol, salloc, srun, sattach, scrontab, sdiag, smd, sprio, sreport, sshare, sstat, strigger) → spur — are created on every install path.
• pkill -f spurd also kills spurctld (substring match). Always pkill -x (exact name).

Accounting

• No separate accounting daemon — upstream folded spurdbd into spurctld. Deploy Postgres BEFORE the controllers (Step 5 precedes Step 6) so each controller's embedded accounting service can connect and migrate on startup.
• Every controller needs network access to Postgres, not just localhost — each spurctld connects directly, so listen_addresses/pg_hba.conf must allow every controller's IP (Step 5), not just ACCT_HOST.
• Migrations run automatically inside spurctld against database_url; a failed migration disables accounting for that controller (scheduling still works, sacct won't) rather than crashing it.
• Default DB creds are spur/spur/spur — fine for a lab, change ACCT_DB_PASSWORD for anything real. Flag this to the user.
• SPUR_WIPE_STATE defaults to false — re-runs and upgrades preserve the Raft job queue and node registrations. SPUR_WIPE_STATE=true resets the Raft job-id counter (job ids restart at 1, upserting onto the same accounting rows); use it only for a fresh install or intentional reinit.
• Rebuild all three binaries together for an upgrade. The daemons share a Raft WAL schema. Pushing a spurctld built from a different tree than its peers can crash it on start (unknown variant … / LogIndex(N) violates) when it reads a log entry it can't parse.
• Changing the controller set needs a wipe (Spur 0.3.0 has no online membership change). Adding/removing/reordering a controller with state preserved leaves openraft with a mismatched on-disk membership. Agents are not Raft members — add/remove them freely.
• A stale dpkg lock (Could not get lock /var/lib/dpkg/lock-frontend) means another apt/unattended-upgrade is running. Wait for it, or clear a genuinely hung apt-get before retrying — don't --force.
• Migrating a pre-merge cluster leaves a stale spurdbd behind if you skip Step 5b — it keeps running (harmlessly) alongside the new embedded accounting until explicitly stopped/disabled/removed.

Spur quirks

• Output file spur-<N>.out goes to the job's WorkDir = the CWD at submit time. Submitting from /tmp writes /tmp/spur-<N>.out; submitting from the SSH user's home writes it there. Do NOT cd into a 0700 dir (e.g. /root/spur) when SSHing as a non-root user — the cd fails and WorkDir silently becomes the user's home. Pin the submit CWD to /tmp for predictability, and search /tmp /home /root ${SPUR_HOME} when hunting for output.
• spur nodes collapses by partition. To verify per-host registration, loop spur show node <name>.
• spur show node <name> does not filter server-side at all — it prints every registered node regardless of the argument. (Tested against Spur 0.3.0; treat the "prefix match" framing as describing symptom, not mechanism.) Always pipe through awk -v n=<name> '/^NodeName=/{p=($0=="NodeName="n)} p' to isolate one node's block — this is required for correctness, not just to break prefix ties.
• spur show job uses JobState=COMPLETED (uppercase). Parse JobState=[A-Z]+.
• Raft port 6821 is hardcoded in spurctld (not a CLI flag). Preflight must include it.
• Harmless log spam invalid transition from Completed to Completed on followers after multi-node jobs — the job actually succeeded.
• Harmless ERROR-level openraft log line on every spurctld restart — Can not initialize last_log_id=Some(...) vote=...:committed. Despite the ERROR severity, this is normal on a restart with existing Raft state (single-node or HA); it doesn't indicate a problem — check spur nodes/job/accounting behavior, not this log line, to judge success.
• Harmless spurd startup warning failed to load spur.conf ... path=/etc/spur/spur.conf — spurd is driven entirely by CLI flags and never actually reads a config file; this warning is always present and doesn't indicate misconfiguration.
• $SPUR_NUM_NODES is not set in the job environment (as of Spur 0.3.0) even in multi-node jobs — only $SPUR_TASK_OFFSET and $SPUR_PEER_NODES are populated. A smoke-test script referencing it will print an empty value; don't treat that as a failure signal.
• A per-job scratch file (e.g. .spur_job_<id>.sh) can be left behind in an agent's WorkDir and, on a shared/reused host, block a later job with a different id from writing its own script (agent rejected job: failed to write job script). The job then sits in COMPLETING forever and the node shows mix instead of idle — there's no automatic timeout/recovery. Fix by spur cancel <jobid> and removing the stale scratch file by hand; teardown (Step 11) does not clean these up since it only removes ${SPUR_HOME} and *.out files, not arbitrary WorkDirs.

Multi-node / HA specifics

• Agent --hostname must match the [[nodes]] name in spur.conf and spur show node <name>. Use hostname -s consistently.
• Pass --hostname and --address explicitly to spurd. Auto-detect picks 127.0.0.1, breaking inter-node dispatch.
• No shared-FS assumption. Each node writes its own spur-<JOBID>.out locally; fetch from every agent.
• HA needs a leader-elected wait, not just port-listening. Loop on no leader elected yet until it clears.
• HA peers list order must be stable across redeploys. node_id is the 1-based position; reordering breaks openraft. To re-order, wipe state on every controller and redeploy.
• -N in a multi-node job must not exceed the agent count, or it stays PENDING.
• Separate-compute HA: controllers NOT in AGENTS must have no spurd. Verify with systemctl is-active spurd → inactive.
• Client-side failover is automatic. spurd --controller and the CLI's SPUR_CONTROLLER_ADDR accept a comma-separated endpoint list and rotate past a dead one, so every agent and each controller's own env lists every controller, not just CONTROLLERS[0]. No VIP/DNS is required for basic failover.

Report back

End the run with:

• Mode + counts (X controllers, Y agents), transport, accounting on/off
• Per-host: install source (binary-src vs installer), systemctl is-active for each daemon, log source (journalctl -u spurctld/spurd, or spurdbd if a legacy pre-merge unit is still present)
• spur nodes output
• HA only: which node_id became leader
• Accounting only: sacct output + DB job count
• Test job IDs + stdout (single, and multi if run)
• Any deviation from this skill — flag it so the skill can be patched