| name | rocs-testbed-slurm |
| description | How to submit SLURM jobs on the SANDS Lab ROCS testbed cluster (KAUST). Covers GPU GRES selection (v100/a100 — p100 is off-policy), QoS caps, conda env activation, multi-node DeepSpeed, preemptible spot jobs, and the watchdog that kills jobs with <15% GPU utilization. Use for any project whose compute_backend is slurm and whose cluster nodes are named `mcnode*`. |
| tags | ["hpc","slurm","rocs-testbed","kaust","gpu","sbatch","conda","mamba"] |
| source | https://sands.kaust.edu.sa/internal/rocs-testbed/slurm-environment/ |
ROCS Testbed — SLURM Reference
This is the SANDS Lab / KAUST internal cluster. Head node mcmgt01, compute nodes mcnode01…mcnode33. Different from IBEX — no module system, no shared filesystem with IBEX.
What experimenters MUST get right
- Activate conda by sourcing its profile script before
conda activate. The bare
conda activate my-env
line will fail with CondaError: Run 'conda init' before 'conda activate' on any
host where the user has not run conda init bash. #!/bin/bash --login alone does
NOT fix this — --login only sources .bashrc / .bash_profile, and those files
only register the conda activate shell function if conda init bash was run
previously. The portable, always-works pattern is:
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "/path/to/project/.env"
Do this in every sbatch script. Shebang can be plain #!/bin/bash.
- Request GPU explicitly via
--gres when the code trains a neural network, runs PyTorch/JAX/TensorFlow, or otherwise benefits from CUDA. Submitting to the mc partition without --gres gets you a CPU-only allocation on a GPU-equipped node — wasted machine-time and badly slow training.
- Pick the weakest GPU that works: V100 > A100 (prefer V100). Only escalate to A100 if the model/batch-size genuinely requires it. Do not use P100 — it exists on the cluster but this project's policy is V100 minimum.
- Set
--time conservatively. Max job length is 14 days; interactive sessions cap at 4h. Jobs with --time > 3 days will soon need to be preemptible.
- Don't push GPU utilization below 15% — any GPU in the job running at <15% for 1h triggers a warning email; 2h consecutive triggers automatic cancellation.
Minimal GPU sbatch template (copy-paste)
#!/bin/bash
set -e
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "<project_env_path>"
python scripts/train.py --config configs/base.yaml
#SBATCH --gres=gpu:v100:1 is the single most important line that experimenters currently forget. If you do ML and don't include it, the node will be assigned but cuda.is_available() returns False and training falls back to CPU — a 20–100× slowdown that can push a 10-minute training run past the pipeline's wait-timeout.
CPU-only sbatch template
For pure data processing / backtesting / plotting that doesn't use a GPU:
#!/bin/bash
set -e
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "<project_env_path>"
python scripts/backtest.py
Do NOT include --gres=gpu:... when you don't use GPU — it burns one of your QoS quota slots (A100=2, V100=8) and blocks a neighbor's real GPU job.
GPU selection guide
| GPU | When to use | Quota (normal QoS) |
|---|
v100 | Default — classical ML, GRU/LSTM, most deep-learning work, moderate-size models (≤200M), mixed-precision | 8 |
a100 | Large models (>500M), long training (days), high-memory batches | 2 |
Note: p100 nodes exist on the cluster (mcnode01, mcnode02) but are not in our allowed GPU set — V100 is the floor. If sinfo / ginfo shows only P100s free, wait for V100 instead of falling back.
For A100 specifically, add --constraint=gpu_a100_80gb or gpu_a100_40gb to pin memory capacity, and --constraint=gpu_sxm if you need multi-GPU NVLink (PCI A100 inter-GPU bandwidth is ~3× slower, and per-card ~10% slower than SXM).
QoS and preemption
- Default QoS =
normal. Caps: A100=2, V100=8 per user (concurrent). P100 has a cap of 8 but is off-policy for this project.
- Low-priority QoS =
spot — no caps, but preemptible by normal-QoS jobs.
- To run a long (>3 day) job, use
--qos=spot with a checkpoint-and-resume signal handler:
srun python3 train_with_checkpoints.py
The ML code must install a SIGUSR1 handler that saves state and exits; SLURM will then auto-requeue the job (same JobID) on a free resource. See the signal.signal(SIGUSR1, ...) pattern in the ROCS docs.
Environment activation — the trap
Wrong (looks right, fails on the cluster):
#!/bin/bash --login
conda activate my-env
python train.py
Right (always works regardless of shell-init state):
#!/bin/bash
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "/path/to/project/.env"
python train.py
The --login flag only sources .bashrc / .bash_profile. Those files
register the conda activate shell function only if conda init bash
was run in them — and it often wasn't. Sourcing profile.d/conda.sh
directly registers the function from a known-good location. conda info --base returns the install prefix of whichever conda is on $PATH,
so this works for miniforge3, Anaconda, Miniconda interchangeably.
conda run --prefix <env> python script.py is another valid pattern —
no activation needed at all. But it spins up a subshell per invocation,
so stick with source + activate when you're running multiple commands
under the same env in one script.
Job lifecycle commands (cluster-specific shortcuts)
| Command | Purpose |
|---|
sbatch script.slurm | Submit batch job |
squeue --me | Your queued/running jobs only |
scancel --me | Kill all your jobs |
srecent | Recent jobs table (alias for sacct -X -o "JobID,Start,End,Elapsed,JobName,AllocTRES,NodeList,State,ExitCode") |
jobstats <jobid> | Text summary (retained forever) |
jobstats <jobid> -g | Grafana dashboard URL (creds rocs/rocs, data retained 15d) |
ninfo | Per-node features + GRES (find which nodes have your GPU type) |
ginfo | GPU pool utilization snapshot |
sinfo | Partitions + node state |
scontrol show job <jobid> gives allocation details including StdOut= / StdErr= paths, useful for the ARK SLURM watcher to poll for progress.
Multi-node / DeepSpeed template
Only use if your training genuinely spans >1 node. Prefer single-node unless memory or throughput demands more. Full template in the upstream docs; the essentials:
#!/bin/bash
set -e
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate "<deepspeed_env_path>"
export NCCL_SOCKET_IFNAME=fabric
export NCCL_DEBUG=INFO
MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
MASTER_PORT=$((${SLURM_JOB_ID} % 16384 + 49152))
srun --wait=60 --kill-on-bad-exit=1 \
torchrun --nproc_per_node gpu \
--nnodes $SLURM_NNODES \
--rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
--rdzv_backend c10d --tee 3 \
train.py --deepspeed --deepspeed_config ds_config.json
Watchdog policies — jobs that get auto-killed
The cluster's job_defense_shield cancels jobs exhibiting waste. For an ARK experimenter this matters because getting cancelled halfway through a backtest wastes the whole run.
- Low GPU utilization: any GPU in your job running at <15% for 1h → warning email. Persists 2h → cancelled.
- Common cause: allocated GPU but CPU-bound data loading, or forgot to move tensors to
cuda.
- Fix: if your code is CPU-heavy, don't request GPU; if it should use GPU, verify
tensor.to("cuda") and pin_memory=True in DataLoader.
Checklist before submitting any sbatch
Verifying a GPU job actually got a GPU
After a job starts, check:
scontrol show job <jobid> | grep -E "TRES|NodeList"
Or from inside the sbatch script, log it:
python -c "import torch; print('cuda_available:', torch.cuda.is_available(), 'device_count:', torch.cuda.device_count())"
When to contact the cluster admins
- Sustained resource needs beyond QoS caps → open a "GPU Reservation Request" issue at
https://github.com/sands-lab/rocs-testbed/issues. Decisions take ~1 week.
- Unexpected cancellations with
State=CANCELLED BY <uid> where uid is NOT your own → that was a watchdog or admin action; jobstats <jobid> shows the reason.
References
