deploy-slurm-cluster

name	deploy-slurm-cluster
description	Deploy a Slurm GPU cluster with DeepOps and prove it works. Use when asked to deploy, install, or rebuild Slurm on one or more GPU servers with this repository.

Deploy a Slurm GPU cluster

Preconditions

• Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver installs may reboot them; no active users or workloads).
• SSH access from the provisioning machine to every host as a sudo-capable user.
• Run everything from the repository root.

Procedure

1. Prepare the environment and verify it:

   git submodule update --init --recursive
   ./scripts/setup.sh
   cp -r config.example config
   

2. Edit config/inventory: put the controller under [slurm-master] and compute nodes under [slurm-node] (a single machine can be both). Set the connection user in [all:vars] if not root.

3. Preflight — must pass before deploying:

   python3 scripts/validation/deepops_doctor.py --remote --json
   

   Fix anything in failures (each check's detail says how) and rerun.

4. Deploy:

   ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml
   

   This installs NVIDIA drivers, builds and configures Slurm, and sets up munge, NFS, and node health checks. Expect roughly 30–60 minutes on a first run.

5. Validate on a cluster node — the success signal is this, not the play recap:

   python3 scripts/validation/validate_slurm.py --json
   

   Require "ok": true with and .