stampede3-submit

name	stampede3-submit
description	Build a Slurm sbatch script for Stampede3 (TACC). Use when the user on Stampede3 asks to "run X on the cluster", "submit a job", "make a batch script", or "request N nodes/GPUs". Covers serial, MPI, OpenMP, hybrid, and GPU (H100/PVC) workloads with the correct partition, ibrun launch, and module setup. For diagnosing a job that already failed, use stampede3-debug.

stampede3-submit — Stampede3 sbatch script builder

Queue reference

Queue	Node	Cores/node	RAM/node	Max nodes/job	Max wall	Max jobs/user	SU/node-hr
skx	Skylake	48	192 GB	256	48 h	40	1
skx-dev	Skylake	48	192 GB	16	2 h	2	1
icx	Ice Lake	80	256 GB	32	48 h	12	1.5
spr	Sapphire Rapids	112	128 GB HBM2e	32	48 h	24	2
nvdimm	Ice Lake large-mem	80	4 TB	1	48 h	2	4
h100	4× H100 SXM5 (96 GB/GPU)	96	1 TB	4	48 h	2	4
pvc	4× Max 1550 (124 GB/GPU)	96	1 TB	4	48 h	2	3

Workflow

1. Gather requirements before writing anything. If the user hasn't said, ask (one short batch of questions):

   • What's the workload? (which executable, MPI? OpenMP? GPU? serial?)
   • How many nodes / ranks / threads / GPUs?
   • Estimated wall time?
   • Which allocation/project? (only if user has multiple — check /usr/local/etc/taccinfo)
   • Inputs/outputs path — should I cd $SCRATCH/<jobdir> first?

2. Pick the partition with this decision flow, using the queue table above for cores/RAM/limits.

   Step 2a — is this a test or a production run?

   • Test / debugging / first time running the script → skx-dev (2 h max, 16 nodes max). Always start here. Faster queue, same SKX hardware.

   Step 2b — GPU or CPU?

   • NVIDIA CUDA / PyTorch / TensorFlow / JAX → h100 (4× H100 SXM5, 96 GB/GPU, NDR IB).
   • Intel oneAPI / SYCL / DPC++ → pvc (4× Max 1550, 124 GB/GPU).
   • No GPU → continue to 2c.

   Step 2c — does one rank need > 256 GB RAM?

   • Yes → nvdimm (single node, 4 TB, 80 cores). Note: only 3 nodes total in this queue and only 1 per job — be sure you actually need it.
   • No → continue to 2d.

   Step 2d — pick the CPU queue by workload character:

   Workload	Recommended	Reason
   Memory-bandwidth bound (sparse linalg, stencil, FFT, CFD)	spr	HBM2e gives 3.5× per-core bandwidth vs SKX
   Memory-capacity bound (RAM/core matters more than BW)	icx	256 GB / 80 cores = 3.2 GB/core (vs SPR's ~1.1)
   Compute-bound, moderate memory, large job (>32 nodes)	skx	Only queue that scales past 32 nodes (up to 256)
   Legacy / well-validated SKX binary	skx	No re-build risk, 1 SU/node-hr (cheapest)

   Tie-break rules:

   • Need > 32 nodes? skx is your only option for CPU.
   • SU budget tight? Cheaper-first: (1) < (1.5) < (2) < (3) < = (4).

Hard "don't" list

• Don't emit --mem (unsupported on Stampede3).
• Don't emit --export=... (breaks env propagation).
• Don't use mpirun/mpiexec/srun to launch MPI — always ibrun.
• Don't use lfs setstripe on $HOME or $SCRATCH (VAST, not Lustre). $WORK is Lustre.
• Don't request more nodes than the queue allows (see the queue table).
• Don't pick max wall time "to be safe" — shorter jobs schedule faster and the 15-min minimum still applies.

Templates

Serial / single-threaded

#!/bin/bash
#SBATCH -J serial
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 02:00:00
#SBATCH -p skx
#SBATCH -o slurm-%j.out

module reset
module load intel/24.0

cd $SCRATCH/myrun
./myprogram input.txt

Pure MPI (SPR example: 4 nodes × 112 ranks = 448)

#!/bin/bash
#SBATCH -J mpi-spr
#SBATCH -N 4
#SBATCH -n 448
#SBATCH -t 04:00:00
#SBATCH -p spr
#SBATCH -o slurm-%j.out

module reset
module load intel/24.0 impi/21.11

cd $SCRATCH/myrun
ibrun ./mpi_program

Pure OpenMP (single SKX node, 48 threads)

#!/bin/bash
#SBATCH -J omp
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 02:00:00
#SBATCH -p skx
#SBATCH -o slurm-%j.out

module reset
module load intel/24.0

export OMP_NUM_THREADS=48
export OMP_PLACES=cores
export OMP_PROC_BIND=close

cd $SCRATCH/myrun
./omp_program

Hybrid MPI + OpenMP (2 ICX nodes, 4 ranks/node × 20 threads = 80 cores/node)

#!/bin/bash
#SBATCH -J hybrid
#SBATCH -N 2
#SBATCH -n 8
#SBATCH --ntasks-per-node=4
#SBATCH -t 03:00:00
#SBATCH -p icx
#SBATCH -o slurm-%j.out

module reset
module load intel/24.0 impi/21.11

# 4 ranks × 20 threads = 80 cores per ICX node
export OMP_NUM_THREADS=20
export OMP_PLACES=cores
export OMP_PROC_BIND=close

cd $SCRATCH/myrun
ibrun ./hybrid_program

H100 GPU (1 node, 4 H100s, one process per GPU)

#!/bin/bash
#SBATCH -J h100
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -t 02:00:00
#SBATCH -p h100
#SBATCH -o slurm-%j.out

module reset
module load cuda

cd $SCRATCH/myrun
ibrun ./gpu_program     # or: python train.py for single-process multi-GPU

Single-process multi-GPU (PyTorch / TF on one H100 node)

#!/bin/bash
#SBATCH -J torch
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 04:00:00
#SBATCH -p h100
#SBATCH -o slurm-%j.out

module reset
module load cuda python3

source $WORK/venvs/torch/bin/activate
cd $SCRATCH/myrun
python train.py        # framework sees all 4 GPUs via CUDA_VISIBLE_DEVICES

PVC GPU (Intel Max 1550, SYCL)

#!/bin/bash
#SBATCH -J pvc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -t 02:00:00
#SBATCH -p pvc
#SBATCH -o slurm-%j.out

module reset
module load intel oneapi

cd $SCRATCH/myrun
ibrun ./sycl_program

Large memory (NVDIMM, 4 TB single node)

#!/bin/bash
#SBATCH -J bigmem
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 12:00:00
#SBATCH -p nvdimm
#SBATCH -o slurm-%j.out

module reset
module load intel/24.0

cd $SCRATCH/myrun
./memory_hog        # can use up to ~4 TB

Job array (parameter sweep, throttled to 20 concurrent)

#!/bin/bash
#SBATCH -J sweep
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 01:00:00
#SBATCH -p skx
#SBATCH -a 1-100%20
#SBATCH -o slurm-%A_%a.out

module reset
module load intel/24.0

cd $SCRATCH/sweep
./run.sh input_${SLURM_ARRAY_TASK_ID}.dat

Job dependency (B starts after A succeeds)

JID_A=$(sbatch --parsable a.slurm)
sbatch --dependency=afterok:$JID_A b.slurm

After submitting

Report back to the user:

• The JobID
• squeue -j <id> snapshot (state, reason, ETA if running)
• Where stdout/stderr will land
• The estimated SU cost: nodes × max_wall_hours × rate (note: actual bill is by elapsed seconds, 15-min minimum)

When the user wants interactive instead

If they're iterating fast, suggest idev instead of repeated sbatch:

idev -p skx-dev -N 1 -n 48 -t 01:00:00
# drops you onto a compute node; run ibrun / programs as usual; `exit` to release