| name | launch-with-slurm |
| description | Reference for launching jobs inside a SLURM allocation via srun (single-node or multi-node). Use whenever work needs to run on allocated compute — from direct user requests ("run on the cluster", "use my running job", "on my allocation", "launch on slurm", "train across N nodes", "dispatch the job") OR from within another skill's workflow (e.g., validate-correctness running validation on the allocation, add-new-model reaching pp=2/ep=2). Covers finding the target job ($SLURM_JOB_ID or squeue, including array-job ID resolution), reading the allocation with scontrol, the srun flags that matter (--jobid, -W 0, -N, -o, --open-mode, --nodelist, --overlap), and gotchas like GPU sharing with live steps and distributed-aware output redirection. |
Launch with SLURM
srun --jobid=<jobid> attaches a step to a running allocation regardless of where it is invoked — login node, ssh'd compute node, or inside the job itself. The scheduler draws nodes from the allocation pool and may pick a node other than the invoking one. Default to it over raw torchrun/bash: srun propagates env vars, handles distributed-aware I/O, and manages signals across ranks correctly — even on a single node with multiple GPUs. The examples/*/launch.sh scripts read SLURM_NNODES/SLURM_NODEID, which srun sets inside a step, so they work unchanged.
Step 1: Find the Job ID
Two sources, one answer — the numeric JobId of the job (e.g. 2193449):
$SLURM_JOB_ID is set — you are inside the job, or in an ssh session on an allocated node that inherits the job environment. Use it as-is; it is already the numeric JobId.- Otherwise — list the user's jobs and pick a RUNNING one: by name if the user named it, ask if several are ambiguous or none is RUNNING, skip PENDING (no compute yet):
squeue -u $USER -o "%.10i %.9T %.6D %.24N %.24j %.10L"
Array jobs: resolve to the numeric JobId. squeue reports elements as <master>_<task> (e.g. 2193448_1), but --jobid wants the element's own numeric JobId — given 2193448_1, it strips the _1, resolves to the array master, and fails with "Job is pending execution" when the master has pending elements:
scontrol show job 2193448_1 | grep -oP '^JobId=\K[0-9]+'
Then read the allocation — don't guess, ask SLURM:
scontrol show job $JOBID
Key fields to extract:
| Field | Example | What it tells you |
|---|
AllocTRES | cpu=208,mem=1860368M,node=1,billing=208,gres/gpu=8 | Node count, GPUs per node, CPUs, memory |
NodeList | orchard-flame-5 or orchard-flame-[3-6] | Which hosts; on most clusters ssh <name> gives direct access |
For a quick remaining-time check, use squeue directly — it returns D-HH:MM:SS without needing to parse timestamps:
squeue -h -j $JOBID -o %L
Before launching anything long-running, compare this against the estimated runtime. If the budget is too tight, surface this to the user instead of launching and getting killed mid-run.
Step 2: Build the srun Command
--jobid=<jobid> — anchor the step to the allocation. Required when $SLURM_JOB_ID is unset or holds a different job; redundant but harmless when it already holds the target.-N <n> — number of nodes to dispatch to. In most training runs this matches PP, but the full parallelism plan and GPUs-per-node determine total nodes (e.g., PP=1 with EP=16 on 8-GPU nodes still needs 2). -N1 borrows one node of a multi-node allocation for probes or single-node tests.-W 0 — wait indefinitely for stragglers after the first task exits. The default behavior terminates remaining tasks shortly after the first one ends, which kills workers that are still cleanly shutting down. Always use -W 0 for training and evaluation runs.-o <file> — stdout redirection. Use this instead of piping through tee. On multi-node, teeing srun output collapses concurrent writes from all ranks. -o is distributed-aware — srun collects output from every rank into the single specified file, preserving the one-command-one-log abstraction. By convention, PithTrain runs log under logging/<descriptive-name>.log.--open-mode=append vs --open-mode=truncate — for resumed training, append preserves history across restarts. Use truncate for fresh runs where overwriting is intended.--nodelist=<hosts> — restrict dispatch to specific nodes. Useful for debugging at a smaller scale (e.g., 4 nodes allocated, but debug with 2 specific ones).--overlap — share the allocation's CPUs, memory, and GPUs with the job's other steps. Without it, a step reserves the whole allocation, so any later step silently pends until it finishes — pass --overlap whenever launching alongside a running step.--gres=gpu:<k> — request a subset of the job's GPUs for the step. By default a step sees every GPU the job holds on the node, so torchrun --nproc-per-node=gpu just works. Sharing means contention: probe nvidia-smi utilization before launching heavy work onto a job whose GPUs are already busy; if it's saturated, surface this to the user instead of piling on.
srun execs the command directly, not through a shell — invoke scripts as bash <script> or ensure the +x bit.
References
