| name | using-drac-clusters |
| description | Use when working on a Digital Research Alliance of Canada (DRAC, Alliance, formerly Compute Canada) HPC cluster such as Narval, Rorqual, Fir, TamIA, Killarney, Vulcan, Nibi or Trillium — including ssh to *.alliancecan.ca failing with "Permission denied (keyboard-interactive)", sinfo/sacctmgr reporting "command not found" over ssh, submitting or monitoring Slurm jobs (sbatch, squeue, sacct, job arrays, GPU allocations), choosing between clusters, staging data into /project or /scratch, or jobs stuck PENDING. |
Using DRAC Clusters
Overview
DRAC clusters are shared, MFA-gated, quota-bound Slurm systems. Three properties
drive almost every mistake: you cannot authenticate on your own, login
nodes are not for computing, and almost nothing is knowable without asking
the cluster — accounts, partitions, GPU types and quotas differ per cluster.
Core principle: discover, don't assume. Every value below has a command that
prints the truth. Run it once per cluster and cache it in the project's notes.
Scope: this skill is the mechanics — authenticating, finding the right
account and gres string, respecting quotas, moving files. For the methodology
of an experiment campaign — cost models and walltime budgeting, how to shape
jobs and arrays, gating a script on the compute it actually got, keeping results
alive through a walltime kill, staged submission — use
running-cluster-experiments.
The MFA gate — read this before anything else
MFA is mandatory on DRAC and an agent cannot complete it. Non-interactive ssh
works only by multiplexing over a ControlMaster socket the user already
authenticated. Before concluding the cluster is down:
ssh -O check <cluster>
With no live master, every command fails identically with
Permission denied (keyboard-interactive). That is not a transient error, a
wrong key, or an outage. Retrying cannot fix it.
Do NOT engineer around the MFA prompt. It is technically possible — pointing
SSH_ASKPASS at a script that auto-selects the push option will get an agent
through, and a subagent did exactly that during this skill's own testing. Do not
do it. MFA exists to put a human decision in front of every new session; a script
that answers it converts the user's phone into a rubber stamp and reproduces the
shape of an MFA-fatigue attack, even when the intent is benign. The user then
approves a push they did not initiate and cannot attribute.
The gate is a stop, not an obstacle. Report the blocker and ask. If the user
later says "go ahead and automate the prompt", that is their call to make
explicitly — never yours to assume from a task that merely needs cluster access.
Two legitimate ways to open the master
A. The user opens it in a REAL terminal. Always available, needs no setup. An
agent-run shell cannot do this, however it is invoked (Claude Code's ! prefix,
a tool call, nohup, a background job): MFA needs an interactive TTY, and
without one ssh falls back to SSH_ASKPASS and fails like this —
Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh_askpass: exec(/usr/X11R6/bin/ssh-askpass): No such file or directory
Permission denied (keyboard-interactive,hostbased)
So the ask is specific: "open Terminal/iTerm and run ssh <cluster>, complete
the MFA prompt." They may exit immediately afterwards — ControlPersist keeps
the master alive once it exists.
B. A GUI askpass, if the user has installed one. With SSH_ASKPASS pointing
at a real dialog program (e.g. theseal/ssh-askpass on macOS) and DISPLAY set,
an agent can start the connection and the user answers a dialog on screen. This
removes the context switch, not the human.
The distinction that matters is not GUI-vs-terminal, it is who answers. A
dialog program ASKS the human; a script that returns a canned answer REPLACES
them. The first is fine. The second is the prohibition above.
With askpass available, ANNOUNCE BEFORE CONNECTING. Say that a dialog is
about to appear and why, then connect. The point is attributability: if the user
can always trace a prompt to something they were just told about, an
unannounced dialog becomes a reliable signal to refuse. Silently summoning
prompts destroys that signal even though every individual prompt is answered by a
human.
Installation note, if asked: install the package but do not run
brew services start. That agent launchctl setenvs SSH_ASKPASS and
SUDO_ASKPASS globally for every GUI app, putting a dialog in front of sudo -A
— far beyond what ssh needs. Export SSH_ASKPASS from the shell profile instead.
A socket that worked an hour ago proves nothing about now; re-check before a
batch of commands rather than discovering it mid-transfer.
Per-cluster ~/.ssh/config that makes this work:
Host narval narval.alliancecan.ca
HostName narval.alliancecan.ca
User <username>
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p.sock
ControlPersist 8h
mkdir -p ~/.ssh/sockets first — if the directory is missing, multiplexing fails
silently and every command re-prompts for MFA. With no master, ssh -O check
prints Control socket connect(...): No such file or directory.
Never compute on a login node
Login nodes are for editing, transfers and sbatch, and staff kill offenders.
Cheap queries (squeue, sinfo, sacct, diskusage_report, ls) are fine;
anything looping over data, training, or running for minutes is not. Use
sbatch, or salloc for interactive work:
salloc --account=<alloc>_cpu --cpus-per-task=4 --mem=8G --time=1:00:00
Discover the cluster, don't assume it
| What | Command |
|---|
| Which cluster am I on | echo $CC_CLUSTER |
| Accounts you may charge | sacctmgr -n show assoc user=$USER format=Account%30 | sort -u |
| Partitions and their time caps | sinfo -h -o "%P %l" | sort -u |
| GPU models actually present | sinfo -h -o "%G" | sort -u |
| Quotas (space and file count) | diskusage_report |
| Your fairshare / why you are queued | sshare -U -u $USER -o Account,RawUsage,LevelFS |
| Job efficiency after it finishes | seff <jobid> (absent on some clusters — fall back to sacct -j <id> -o Elapsed,MaxRSS,TotalCPU) |
How much actually varies — six clusters, same day, same user
This is why the principle is "discover, don't assume". Every value below was read
off the cluster with the commands above; none is inferred from another cluster.
| Narval | Rorqual | Fir | TamIA | Killarney | Vulcan |
|---|
| Account | def-x_cpu/_gpu | def-x_cpu/_gpu | def-x_cpu/_gpu | aip-x, no suffix | aip-x, no suffix | aip-x, no suffix |
| Max walltime | 7d (b5) | 7d (b5) | 28d (b6, CPU); 7d GPU | 24h (b3) | 7d (b5) | 7d (b5) |
| GPU partitions | bygpu+bynode | bygpu+bynode | bygpu+bynode | bynode only | gpubase_<model>_bN | bygpu+bynode |
| CPU partitions | yes | yes | yes (+cpularge) | yes | none — GPU-only | yes |
| GPU gres | gpu:a100, MIG a100_3g.20gb | gpu:h100, MIG nvidia_h100_…_3g.40gb | gpu:h100 ×4, same MIG names | gpu:h100 ×4, gpu:h200 ×8 | gpu:h100 ×8, gpu:l40s ×4 | gpu:l40s ×4 + shard:l40s:16 |
/home | 50 GB / 500K | 50 GB / 500K | 48 GiB / 500K | 25 GiB / 250K | 50 GiB / 500K |
Three of those cells contradict what a reasonable person would generalise from
the first three columns, and each is its own trap:
Time buckets do not stop at b5. General-purpose clusters may expose _b6
at 28 days for CPU, and preempt partitions running to 122 days. Reading
"b5 = the maximum" off one cluster under-uses another by 4x. Conversely an AI
cluster may stop at b3. Always read sinfo -h -o "%P %l".
Partitions are not always named bygpu/bynode. At least one cluster names
them by GPU model — gpubase_h100_b3, gpubase_l40s_b3 — so a script that
greps for bygpu finds nothing and a --partition=gpubase_bynode_b3 copied
across is rejected outright. On such a cluster you select hardware by partition,
not only by --gpus=.
Some clusters have no CPU partitions at all. A GPU-only cluster will reject
CPU-only preprocessing, so the "run it in salloc instead of on the login node"
advice has nowhere to land there — stage that work on a different cluster.
Fractional GPUs come in two unrelated flavours. MIG slices appear in the gres
string as _1g. / _3g. variants; shard: is Slurm's time-sharing of a
whole GPU (e.g. shard:l40s:16 = 4 shards per card). Requesting a shard gives
you a card shared with other jobs, not a partitioned one — fine for small
inference, wrong for a benchmark you intend to time.
Inode quotas differ by four orders of magnitude — 250K on one cluster against
105M on another, because they are not the same filesystem. A pipeline that is
fine on one will wedge a group's quota on another.
Some clusters set up Slurm only in a LOGIN shell
On most clusters ssh <cluster> 'sinfo' just works. On at least one it does not,
and the failure is loud in a misleading way:
bash: line 1: sinfo: command not found
bash: line 2: sacctmgr: command not found
with $CC_CLUSTER, $SCRATCH and $PROJECT all empty. The cluster is
fine — the Slurm binaries and site environment come from the login profile, and
a non-interactive ssh <host> 'cmd' never sources it. Every diagnostic you would
reach for to check whether the cluster is healthy fails at once, which reads
exactly like a broken or half-provisioned account.
Fix: force a login shell.
ssh <cluster> 'bash -lc "sinfo -h -o \"%P %l\" | sort -u"'
Never conclude an account is unprovisioned from command not found. Retry
through bash -lc first, and note per cluster whether it is needed — it changes
how every scripted command against that host must be written, including the ones
inside your job-submission wrappers. An empty $SCRATCH is especially dangerous:
a path built as "$SCRATCH/data" silently becomes /data.
Accounts often but NOT always carry a resource suffix (<alloc>_cpu,
<alloc>_gpu). Where they do, requesting a GPU under _cpu fails and CPU work
charged to _gpu burns a scarcer allocation. Where they don't (AI clusters using
aip- allocations), one account covers both. Always run sacctmgr; never copy an
account string between projects or clusters.
Never construct user paths — use $HOME and $SCRATCH. Some clusters shard
both by first letter, and the constructed form does not merely differ, it does
not exist:
narval/rorqual HOME=/home/$USER SCRATCH=/scratch/$USER
tamia HOME=/home/<u>/$USER SCRATCH=/scratch/<u>/$USER
where <u> is the first letter of your username
ls /scratch/$USER on TamIA returns No such file or directory. ($PROJECT is
not set on any of them — get the project path from diskusage_report or ls /project/.)
GPU gres strings are NOT consistent across clusters — always read them from
sinfo -h -o "%G". Two clusters, same year, same generation of hardware:
narval gpu:a100 gpu:a100_1g.5gb gpu:a100_3g.20gb # short MIG names
rorqual gpu:h100 gpu:nvidia_h100_80gb_hbm3_1g.10gb # fully-qualified
A --gpus= string copied from another cluster's job script fails, and MIG slices
(the _1g. / _2g. / _3g. variants) are fractional GPUs — request one by
accident and you get a sliver of a card. Ask for a whole GPU explicitly.
Time buckets decide your queue. Partitions are binned by walltime: _b1 ≤3h,
_b2 ≤12h, _b3 ≤24h, _b4 ≤3d, _b5 ≤7d. Asking 13h instead of 12h moves you
to a scarcer partition and can cost days of waiting — round --time down into a
bucket.
How FAR the buckets go is a hard cap that differs per cluster. General-purpose
clusters run to b5 (7 days); AI clusters may stop at b3, making 24h the
maximum job length — a 48h job is not slow to schedule there, it is impossible,
and long work must checkpoint and requeue. Always check:
sinfo -h -o "%P %l" | sort -u
Filesystems: file COUNT is the quota that bites
| Purpose | Backed up | Watch |
|---|
/home | code, configs | yes | small (~50 GB), low file cap |
/project | shared datasets, results | yes | file-count quota, often ~500K |
/scratch | large, regenerable, in-flight | no | purged (commonly 60 days) |
Space is rarely the limit; inodes are — millions of small files wedge a
group's /project while it looks half empty. Put re-fetchable data and
high-file-count output on /scratch.
Copying into /project must preserve group ownership:
rsync -a --no-g --no-p --stats src/ <cluster>:/project/<group>/<user>/<proj>/src/
Without --no-g --no-p, rsync imposes local ownership and breaks the setgid
group inheritance /project relies on, locking out collaborators. This fails
silently — verify after every transfer:
ssh <cluster> "cd <projdir> && find . -type f ! -group <group> | wc -l"
Use --stats, not --info=stats1, for a summary that prints reliably.
Submitting and monitoring
Calibrate before submitting an array. Run ONE task, read its elapsed time,
then set --time with headroom. A 200-task array on a guessed walltime either
wastes an allocation or dies at 99%.
squeue -u $USER -o '%.14i %.22j %.8T %.10M %.10l %R'
sacct -X -S <YYYY-MM-DD> -o JobID%16,JobName%22,State,Elapsed,ExitCode
sacct -j <id> -X -n -o State | sort | uniq -c
-X gives one row per job instead of per job step.
Never pipe a running job's output through tail, head or grep — they
buffer, so the file stays empty and you lose all visibility into something you
already started. Redirect to a file and read the file.
One failed task in an otherwise healthy array is usually a bad node. Look for
hardware-flavoured errors (e.g. CUDA driver version is insufficient for CUDA runtime version) before suspecting your code; resubmit that index alone with
sbatch --array=<n> <script>.
Common mistakes
| Mistake | Consequence |
|---|
Retrying ssh after Permission denied (keyboard-interactive) | Cannot succeed; the user must re-authenticate MFA |
| Asking the user to authenticate via an agent-run shell | No TTY, so ssh_askpass fails; it must be a real terminal |
| Assuming a working socket stays working | Commands fail mid-session; re-check with ssh -O check |
| Copying an account string between projects or clusters | Job rejected; the _cpu/_gpu suffix does not exist everywhere |
Copying a --gpus= string between clusters | Rejected, or a MIG sliver instead of a whole GPU |
Copying a --partition= string between clusters | Some name partitions by GPU model (gpubase_h100_b3), so bygpu/bynode do not exist there |
Requesting shard: thinking it is a whole GPU | You get a time-shared card; timings are meaningless |
Concluding an account is unprovisioned from sinfo: command not found | Slurm may only be on the PATH in a LOGIN shell — retry via bash -lc |
Building a path from an empty $SCRATCH in a non-login shell | "$SCRATCH/data" silently becomes /data |
Assuming b5/7d is the longest bucket anywhere | Some clusters expose b6 at 28 days; you under-use them by 4x |
| Assuming every cluster has CPU partitions | GPU-only clusters reject CPU-only work outright |
Assuming --time=48:00:00 will just queue slowly | On a b3-capped cluster it is unsubmittable; checkpoint instead |
Hardcoding /home/$USER | Wrong on clusters with sharded homes; use $HOME |
--time just over a bucket boundary | Scarcer partition, far longer queue |
rsync into without |
Red flags — stop and check
- About to script
SSH_ASKPASS, or otherwise auto-answer an MFA prompt → stop
and ask. Completing the task is not worth silently spending the user's
second factor
- About to run anything longer than a few seconds over ssh → that is a login node
- About to say the cluster is down → run
ssh -O check first
- About to say an account is unprovisioned because Slurm commands are missing → retry through
bash -lc
- About to reuse a partition name, bucket ceiling or gres string from another cluster → read
sinfo on THIS one
- About to submit >20 tasks without having timed one → calibrate
- About to report a job's progress from a buffered pipe → you are reading nothing
