| name | alliancecan |
| description | Alliance Canada Slurm clusters — default to L40S on Killarney, jobs, storage, modules, paths, and account pitfalls. Use when editing Slurm scripts, cluster setup, or HPC workflows for docs.alliancecan.ca systems. |
Alliance Canada HPC
Apply when working on Slurm job scripts, cluster setup, paths, or GPU requests for Alliance national systems (Killarney, Fir, Narval, Nibi, Rorqual, Trillium, etc.).
Default GPU for this repo: L40S (always prefer this first)
For Killarney, treat NVIDIA L40S as the default GPU in new or edited Slurm scripts unless the user explicitly asks for H100/A100 or you know they need 80 GB VRAM.
Why: L40S usually has a much shorter queue than H100 performance tiers. H100 is overkill for many PoC / IPPO / probe workloads and can sit in PD for hours.
How to request L40S (standard pattern)
In the job script, use --gres — do not put H100-only partitions (gpubase_h100_*) together with L40S gres (incompatible).
Submit from the repo directory:
sbatch --account=<your-ccdb-group> your_job.sh
sbatch your_job.sh
Verify what Slurm recorded (after submit):
scontrol show job=<JOBID> | egrep -i 'ReqTRES|TRES=|MinMem'
You should see l40s in the GRES line, not h100.
When to use H100 instead (opt-in)
Use H100 only when the workload needs large GPU memory (e.g. huge batches, very large models) or the user asks for the performance tier. Typical Killarney pattern:
Match b1…b5 partition letter to max wall time allowed on that queue (sinfo -o "%P %G %l"). Never mix gpubase_h100_* with --gres=gpu:l40s:1.
Slurm account (read this first — common failure mode)
- Every job needs
#SBATCH --account=<group-name> where <group-name> is the Group Name from CCDB → My Projects → My Resources and Allocations. It is not a display title; it must match Slurm exactly.
- If
sbatch says Invalid account and lists allowed accounts (e.g. aip-boyuwang), use one of those strings verbatim.
- Never instruct anyone to run
sed that replaces a real account with a placeholder like your-ccdb-group or YOURGROUP — the job will fail. Either keep a real example account in the script or tell them to set the real name from CCDB / sbatch’s error list.
- Optional:
export SLURM_ACCOUNT=aip-boyuwang and export SBATCH_ACCOUNT=$SLURM_ACCOUNT in ~/.bashrc, or override per job: sbatch --account=aip-boyuwang script.sh.
Agent instructions: pull / prep / submit command blocks
When the user asks how to run a Slurm job on Alliance (especially Killarney), give all login-node commands in a single fenced bash block they can copy-paste in one shot.
Rules:
- One block for the full workflow (e.g.
cd, git pull, one-time clones, sbatch, monitor). Do not scatter commands across paragraphs unless the user asked for explanation only.
- Comment out lines that are not required for this request: other datasets,
--smoke-test, --skip-mmpd-train, alternate scripts, optional walltime overrides, duplicate submit examples, etc. Use # at line start.
- Leave uncommented only what the user needs for the job they named (script path, dataset(s), full vs smoke if they specified).
- Assume repo root is
$SCRATCH/ts-sandbox on Killarney unless they gave another path; never cd $SCRATCH/$USER.
- If prep is one-time (e.g.
git clone into temp/MMPD), include it in the same block above sbatch.
Example shape (adapt paths/flags to the actual script):
cd "$SCRATCH/ts-sandbox"
git pull
mkdir -p temp
./slurm_example.sh
$SCRATCH and $USER (second common failure mode)
On Killarney, $SCRATCH is typically already your user scratch directory (e.g. /scratch/ccao87). It is not /scratch with a missing $USER segment that you must add by hand.
- Wrong:
cd $SCRATCH/$USER when $SCRATCH=/scratch/ccao87 → becomes /scratch/ccao87/ccao87 → No such file or directory.
- Right:
cd $SCRATCH and clone/checkout the repo there, e.g. $SCRATCH/drc-sokoban-ma (same pattern as ts-sandbox using $SCRATCH/ts-sandbox).
Repo job scripts in this project should resolve the working tree like:
if [ -d "$SCRATCH/drc-sokoban-ma" ]; then
export PROJECT_ROOT="$SCRATCH/drc-sokoban-ma"
elif [ -d "$HOME/drc-sokoban-ma" ]; then
export PROJECT_ROOT="$HOME/drc-sokoban-ma"
else
echo "ERROR: clone repo to \$SCRATCH/drc-sokoban-ma (or \$HOME fallback)"
exit 1
fi
Accounts and Slurm basics
- Scheduler: Slurm only. No compute on login nodes except tiny tasks (~≤10 CPU-minutes, ~≤4 GB RAM). Everything else:
sbatch, salloc, srun.
- Minimum directives: Always set
#SBATCH --time=.... Add --mem or --mem-per-cpu on general-purpose clusters (default can be very small per core). #SBATCH lines must come before any shell commands in the script.
Where to put files (storage hygiene)
- HOME (
~): Small quota; keep source, job scripts, tiny configs. Not for large datasets or heavy I/O.
- SCRATCH: Large, fast for big sequential I/O; not backed up; old files may be purged (e.g. 60-day policy—check current docs). Use for checkpoints during runs, bulk output, datasets you can re-fetch, and clone/run the repo where policy requires (Killarney: no GPU work from
/home — run from $SCRATCH/...).
- PROJECT (
$PROJECT / ~/projects/...): Shared by the allocation group, larger quota, backed up; intended for relatively static shared data—frequent churn hurts tape backup. For your artifacts, use a per-user subdirectory: $PROJECT/$USER/<app>/ (venv, checkpoints, wandb, copied datasets), not the group root as a personal scratch pad.
$SLURM_TMPDIR: Per-job local disk on the compute node; great for many small files and ephemeral shuffles; deleted when the job ends.- Python/R packages: Often not full Lmod modules; use Alliance Python + pip/wheel docs, or install into your space (venv under
$PROJECT/$USER/... or similar).
- Modules: Start job scripts with
module purge then load what you need; prerequisites matter—use module spider <name>/<version> to see the load chain. Avoid relying on whatever was loaded in the interactive shell.
Per-user working space (important): Treat group project space as shared infrastructure. Keep your working data, venvs, and experiment outputs under $PROJECT/$USER/... and/or $SCRATCH/$USER/... (only when that path is valid on your site). Do not stash personal experiments or venvs directly under $PROJECT/ without a /$USER/ (or agreed team) path—avoids quota fights and policy issues.
Repo-local results/ on the cluster (Slurm + Python)
When you add or edit Slurm job scripts or Python that runs on Alliance login/compute nodes, put all run outputs the job creates—logs, checkpoints, generated datasets, exports—under ./results/ relative to the submission directory (the directory you cd to before sbatch, i.e. SLURM_SUBMIT_DIR in the batch environment). Do not anchor paths off the spool copy of the script.
Use exactly four buckets at the top level of results/—no extra subdirectories unless the user explicitly asks:
| Path | Use |
|---|
./results/{descriptive-run-name}/logs/ | Job/Slurm logs, traces, text or CSV that behave like logs |
./results/{descriptive-run-name}/ckpts/ | Model checkpoints (.pt, .ckpt, .safetensors, …) |
./results/{descriptive-run-name}/datasets/ | Generated or copied data files |
Naming (one file per artifact, flat inside each bucket):
./results/{logs|ckpts|datasets}/{MM-DD}-{last-3-chars-of-$SLURM_JOB_ID}-{short-descriptive-slug}.{ext}
Example: ./results/logs/05-02-847-etth1-train.log, ./results/ckpts/05-02-847-best.pt
After cd "$SLURM_SUBMIT_DIR" (or equivalent), compute a stem and pass it into Python/CLI via env or flags:
STEM="$(date +%m-%d)-${SLURM_JOB_ID: -3}-etth1-latent"
mkdir -p ./results/logs ./results/ckpts ./results/datasets
Git: keep results/logs/, results/ckpts/, and results/datasets/ gitignored in the repo (nested submit dirs: ignore **/results/logs/, etc., or the whole results/ tree).
Slurm: one combined log file (stdout + stderr). Do not leave the default split of *.out and *.err if you can avoid it.
- Preferred: set
#SBATCH --output and #SBATCH --error to the same file path under ./results/logs/—Slurm merges both streams into that file. Slurm expands %j, %x, etc. in those paths; it does not run shell, so for the MM-DD-…-${SLURM_JOB_ID: -3}-… pattern either:
- use
exec >>"$LOG" 2>&1 early in the script (right after mkdir and LOG=…), with #SBATCH --output=/dev/null and #SBATCH --error=/dev/null so only that file receives output, or
- use identical
-o/-e with %j / %x if a simpler name is enough.
Resolving $PROJECT in job scripts (ts-sandbox pattern)
If $PROJECT is empty in the batch environment:
if [ -z "$PROJECT" ] && [ -d "$HOME/projects" ]; then
shopt -s nullglob
_m=("$HOME"/projects/def-* "$HOME"/projects/aip-*)
shopt -u nullglob
if [ "${#_m[@]}" -gt 0 ]; then
export PROJECT=$(readlink -f "${_m[0]}")
fi
fi
Then set storage, e.g. STORAGE_ROOT="$PROJECT/$USER/drc-sokoban-ma" for checkpoints, wandb, venv.
Killarney — hardware reference (verify with sinfo)
This cluster is common for this repo. Verify live with sinfo -o "%P %G %l" and scontrol show node | grep -i gres—names and partitions change.
-
L40S (default tier for scripts in this repo): Standard compute; request with #SBATCH --gres=gpu:l40s:1. Example scripts: slurm_ci_latent_etth2.sh, slurm_ma_tom.sh (default), slurm_latent_experiment.sh.
-
H100 (optional, heavy jobs): Dell XE9680-class nodes, 8× H100 SXM 80GB per node. Use #SBATCH --partition=gpubase_h100_b* + #SBATCH --gpus-per-node=h100:N — not mixed with L40S gres.
-
Code location: Killarney must not run GPU work from /home; keep a checkout under $SCRATCH/<repo-name> (e.g. $SCRATCH/drc-sokoban-ma, same idea as $SCRATCH/ts-sandbox in the other repo).
-
Account prefix: Allocations may show as aip-... on Killarney vs def-... elsewhere—use the CCDB Group Name for --account.
Other clusters (quick reference)
Repo comments (slurm_pipeline.sh): Narval → e.g. A100; Fir / Nibi / Rorqual → e.g. H100-style requests. Always match --account to an allocation valid on that cluster (RAPs are not always portable). Default to smaller / general GPU when unsure, not the largest SKU.
Software modules (typical ML stack)
Example stack used in this repo’s Slurm scripts:
module purge || true
module load StdEnv/2023
module load python/3.11
module load cuda/12.2
module load cudnn/8.9
Load CUDA/cuDNN versions compatible with your PyTorch build. If a module fails, use module spider to resolve prerequisites. Docker is not available; Apptainer is (module load apptainer).
module purge and set -e (common silent failure)
On Alliance nodes, module purge exits non-zero (often code 2) when sticky modules
(CCconfig, gentoo, compiler stack) refuse to unload. With set -e or set -euo pipefail
this kills the job immediately — only the module warning appears in .err, no Python
output, sacct shows FAILED ExitCode=2:0.
Always write module purge || true, never bare module purge in scripts using set -e.
Python virtual environments on compute nodes (critical)
Never keep the active venv on /scratch or /project for a running job.
Both are parallel (Lustre/GPFS) filesystems — fast for large sequential I/O but
extremely slow at reading thousands of small files. import torch touches hundreds of
.so and .py files; on a cold compute node this alone can take 5–15 minutes. A 15-
or 20-minute smoke test times-out before a single line of user code runs.
The fix: rebuild the venv on $SLURM_TMPDIR at job start
$SLURM_TMPDIR is a fast node-local NVMe SSD scoped to your job. Imports from there
take seconds. Canonical Alliance Canada pattern:
virtualenv --no-download "$SLURM_TMPDIR/env"
source "$SLURM_TMPDIR/env/bin/activate"
pip install --no-index --upgrade pip
pip install --no-index torch numpy scikit-learn
pip install --no-index wandb
--no-download skips fetching a new virtualenv wheel. --no-index uses only the pre-built
wheels exposed by module load python/3.11. Check availability: avail_wheels <pkg> on a
login node. Do not cp -r an existing venv — absolute paths in activation scripts break.
Persistent venv jobs: fail fast, don't rebuild in workers
Some repo scripts intentionally reuse a prebuilt shared venv such as
$SCRATCH/ts-sandbox/results/venv to avoid per-job setup. For those scripts, do not
silently fall back to building a new venv inside each Slurm worker when the shared venv is
missing. We have seen Killarney nodes where the shared venv or Lmod init was flaky
(Too many levels of symbolic links), causing repeated pip retries and mixed torch versions.
Pattern for these scripts:
CFG_VENV="${CFG_VENV:-$SCRATCH/ts-sandbox/results/venv}"
[[ -x "$CFG_VENV/bin/python" ]] || {
echo "ERROR: required venv missing: $CFG_VENV" >&2
exit 1
}
sbatch --export=ALL,CFG_VENV="$CFG_VENV" ...
Inside the worker, source the requested venv and verify imports/CUDA. If it is not visible,
exit with a clear message and resubmit excluding that node, or rebuild the venv from the login
node. Avoid ad hoc pip install fallbacks in compute workers unless the script is explicitly
designed to build a $SLURM_TMPDIR venv from Alliance wheels only.
wandb must use --no-index (PyPI sdist → Go → silent pip death)
If a job script does pip install wandb or pip install "wandb>=…" without --no-index, pip may pull a PyPI source distribution. Recent wandb builds run metadata generation that tries to compile wandb-core in Go (Building wandb-core Go binary…). Compute nodes typically do not have go, so pip errors with Did not find the 'go' binary. With set -e in the same shell, the batch script exits immediately after venv setup — logs look “empty”: only the first banner lines and [setup] Building venv on $SLURM_TMPDIR, then nothing until the job ends.
Fix: install wandb from the Alliance wheel cache only: pip install --no-index wandb (Alliance docs and internal wiki: same pattern). Pure-Python extras can use PyPI if needed; do not let wandb resolve from PyPI inside Slurm jobs.
Related: In bash unquoted heredocs used to write job bodies on the login node, backticks in comments still run command substitution (e.g. `import torch` can invoke ImageMagick’s import), and $SLURM_TMPDIR in comments still expands under set -u on the login node. Escape as \$(…) / \$VAR or avoid backticks in those heredocs.
Pin PyTorch on Killarney (pip install --no-index torch unpinned)
After module load cuda/12.2 cudnn/8.9, pip install --no-index torch (no version pin) may pull torch 2.12+ from the Alliance wheelhouse. On Killarney L40S this often fails CUDA init:
CUDA initialization: The NVIDIA driver on your system is too old (found version 12080)
Then torch.cuda.is_available() is False. Training code that does device = cuda if torch.cuda.is_available() else cpu runs on CPU while Slurm still allocated an L40S — nvidia-smi shows 0 MiB / 0% GPU, the Python process shows high CPU (e.g. 400%+), and logs look stuck for hours (e.g. no lines between “Starting Diffusion HP search” and “Trial 0/3” because Optuna only logs when a trial finishes).
Fix — pin 2.11 and fail fast:
module load cuda/12.2 cudnn/8.9
pip install --no-index 'torch==2.11.0+computecanada' numpy scipy pandas scikit-learn wandb ...
python -c "import torch; assert torch.cuda.is_available(), 'CUDA required'; print('torch', torch.__version__, 'gpu', torch.cuda.get_device_name(0))"
If the assert fails, do not let the job train — fix the wheel pin or modules, then resubmit. scancel long-running jobs that passed setup but never use the GPU.
Symptom of the slow-import bug: job output has only the first few echo lines, then
TIMEOUT in sacct. No Python output. Venv is on /scratch.
Smoke-test wall time: even with $SLURM_TMPDIR, request ≥20 min — pip install
from the wheel cache takes 3–5 min; after that imports are <1 min.
Patterns from this repository
#SBATCH --account=aip-boyuwang — example only; replace with your real CCDB group name if different, never with a fake placeholder string.- Venv and bulky data:
$PROJECT/$USER/<app>/venv (and checkpoints/results under the same tree)—per-user under project, not the bare group directory.
slurm_ma_tom.sh: defaults to L40S (--gres=gpu:l40s:1); H100 only via explicit sbatch overrides (see script header).slurm_unet_fullvar.sh (ts-sandbox): L40S for --smoke-test, H100 + gpubase_h100_b4 for full runs.PYTHONUNBUFFERED=1 and python -u help with Slurm log latency (buffering).
Gotchas (see also AGENTS.md — Notes space / repo context — and .ai/cluster-paths.md)
ls ~/projects/def-* under set -euo pipefail (silent job death): Many job scripts use set -euo pipefail and then FIRST=$(ls -d "$HOME"/projects/def-* ... 2>/dev/null | head -1). If ~/projects exists but no def-* / aip-* match, ls exits with status 2. With pipefail, the pipeline fails → set -e aborts the entire batch in a few seconds. sacct shows State=FAILED, ExitCode=2:0, .err may only show module purge noise, .out stays nearly empty. Fix: use shopt -s nullglob and a bash array over the globs (see snippet above), or append || true to the assignment in a way that cannot still fail the pipeline—do not rely on ls with possibly unmatched globs in a piped command.x="$([[ ... ]] && echo y)" under set -e (instant FAILED ExitCode=1:0, no log): Bash assignments propagate the exit status of the last command substitution to set -e. When the [[ ]] test is false, && short-circuits and the cmd-sub exits 1 — the whole assignment "fails" and the job dies before exec >>"$LOG" runs. Same symptoms as the BASH_SOURCE gotcha: Elapsed=00:00:00, ExitCode=1:0, no repo log, no Slurm .out/.err if both go to /dev/null. Fix: build the suffix with a separate if statement, not an inline cmd-sub: SUF=""; if [[ "$SMOKE" -eq 1 ]]; then SUF="-smoke"; fi; STEM="...${SUF}". Note: cmd $([[ x ]] && echo y) as a command argument is safe — only assignments propagate cmd-sub exit to set -e. Also open the run log before any post-cd check that can exit 1, so the next failure leaves a tail-able file.BASH_SOURCE[0] / SCRIPT_DIR in sbatch scripts (instant FAILED, ~0s, empty log): Slurm runs a spool copy (/cm/local/apps/slurm/var/spool/job<ID>/slurm_script), not the file under $SCRATCH/.... If you set SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")" once at the top and later source "$SCRIPT_DIR/slurm/helper.sh" (or similar) in the batch body, that path is the spool dir → file not found → set -e exits before exec >>"$LOG". sacct shows ExitCode=1:0 and ~0 elapsed; with #SBATCH --output=/dev/null there is often no Slurm .out/.err and no repo log — looks like an “instafail” with nothing to tail. Fix: after cd "$SLURM_SUBMIT_DIR", set SCRIPT_DIR="$SLURM_SUBMIT_DIR" (or REPO_ROOT="$SLURM_SUBMIT_DIR") before sourcing helpers or resolving repo data; keep the login-node BASH_SOURCE path only for the submit branch ([[ -z "${SLURM_JOB_ID:-}" ]]) when calling sbatch /path/to/repo/script.sh. Guard: [[ -f "$SCRIPT_DIR/slurm/helper.sh" ]] || { echo "missing helper"; exit 1; }.- Empty/broken venv on cluster: If jobs fail with missing
torch, recreate or repair $PROJECT/$USER/.../venv (or delete and let the Slurm script reinstall).
- Imports: Run Python as
python -m package.module from the repo root.
- Slurm output buffering can make logs look “stuck”; use unbuffered Python or interactive
salloc to debug.
module purge in jobs — always write module purge || true; bare module purge exits non-zero on sticky modules and kills the job when set -e is active (see Software modules section above).sbatch --wrap uses /bin/sh, not bash: --wrap="source ..." fails with source: not found because /bin/sh on Alliance nodes is dash, which uses . not source. Always write a proper #!/bin/bash script and pass it to sbatch — never use --wrap for anything involving source, bash arrays, or [[ ]].sbatch from scripts: Do not submit thousands of jobs at once; prefer arrays or spacing submissions—Alliance warns this can harm Slurm.sbatch stdin vs script file: Prefer sbatch /path/to/job.sh (real file on disk, #!/bin/bash) over piping a heredoc into sbatch. Some sites log or handle stdin batch scripts inconsistently; file-based submission matches working patterns in this repo (e.g. self-submit scripts).- Stale script on cluster: If
squeue still shows H100/64G after switching to L40S, git pull on the cluster clone and resubmit — old #SBATCH lines are baked in at submit time.
- Unpinned
torch on Killarney → silent CPU training: pip install --no-index torch may install 2.12+; CUDA init fails, cuda.is_available() is false, GPU idle while training hammers CPU for hours. Pin torch==2.11.0+computecanada and assert torch.cuda.is_available() right after venv setup (see Python venv section).
signatory.logsignature segfault on Killarney (training dies mid-epoch): After venv setup, logs show latent_rep=logsignature then Segmentation fault from Python (often epoch 1–2). Truncated signature latents are stable on the same nodes. Slurm sets --latent-rep signature and DATALOADER_WORKERS=0 in slurm_signature_diffusion_tune.sh. Opt-in logsignature only with SIGDIFF_LATENT_REP=logsignature once you have verified it on an interactive GPU (salloc + one batch). Do not resume mixed studies: use a new --study-name or Optuna DB if you switched latent type.reformer-pytorch not in wheelhouse (signature venv dies at setup): cluster/setup_signature_cluster_venv.sh used to always pip install --no-index reformer-pytorch==1.4.4 …. On Killarney that package is not in the Alliance wheel cache; four retries then set -e kills the job before Python runs. Logs show only [setup] pip failed … reformer-pytorch and no training output. Log-signature diffusion does not import reformer — set export SIGNATURE_VENV_PROFILE=diffusion in slurm_signature_diffusion_*.sh (installs torch + signatory + optuna only). iTransformer signature-MSE still needs SIGNATURE_VENV_PROFILE=full (default): build reformer on the login node with BUILD_SHARED_VENV=1 (PyPI allowed) or SIGNATURE_USE_SHARED_VENV=1, because compute nodes often block outbound PyPI. Check profile in verify line: venv ok: profile diffusion ….
Official docs
Prefer docs.alliancecan.ca for authoritative quotas, partition names, and policy updates—this skill is a condensed assistant checklist, not a substitute for current site documentation.
