| name | nersc |
| description | How to work productively at NERSC. Covers Slurm job submission and QOS selection (salloc/sbatch/srun, --account, --qos, --constraint), filesystem strategy (HOME/CFS/PSCRATCH/COMMUNITY) and quotas, login-vs-compute discipline, NERSC-specific gotchas (mandatory --account, GPU repo `_g` suffix, $PSCRATCH purge policy, srun for everything that runs on compute, login-node CPU caps), and module/conda environment setup. The current flagship system is Perlmutter (CPU = AMD Milan, GPU = 4× A100/node); the durable conventions carry across system upgrades. DESC NERSC repos: m1727 (CPU), m1727_g (GPU). Use this skill whenever $NERSC_HOST is set, the user runs Slurm commands, touches /global/cfs or /pscratch, works with module load, or asks about NERSC-specific workflows. |
Working at NERSC
You're helping a user at NERSC, the DOE supercomputing center at LBL. The current flagship is Perlmutter (CPU + 4× A100 GPU) and the examples below are written for it; node names, constraints (-C cpu/-C gpu), and the $PSCRATCH mount may change with the next system, but the NERSC-wide conventions — account/repo model, QOS naming, CFS, IRIS, JupyterHub — carry across.
Hard rules: never run real work on a login node, always pass --account=<repo> (Slurm jobs without it die), prefer $PSCRATCH for big I/O, and use srun to launch anything inside an allocation that should land on compute.
Action-oriented; jump to the section that fits. Bundled scripts in scripts/, deeper reference in references/.
Accounts and paths
| Repo | For |
|---|
m1727 | CPU jobs (default) |
m1727_g | GPU jobs — must pass -A m1727_g explicitly |
| Path | Var | Use |
|---|
/global/homes/<l>/<user> | $HOME | Code, dotfiles, venvs (40 GB / 1M inodes) |
/pscratch/sd/<l>/<user> | $PSCRATCH / $SCRATCH | Big I/O, intermediates — purged after ~8 wk inactivity |
/global/cfs/cdirs/<project>/users/<user> | — | Persistent project-shared output |
/global/common/software/<project>/envs/<name> | — | Shared conda/venv envs (fast import) |
Rule of thumb: inputs from CFS, big I/O on $PSCRATCH, code in $HOME, keepers in $CFS/<project>/users/<user>.
Where am I? (always do this first)
scripts/where_am_i.sh
hostname
echo "JOB=$SLURM_JOB_ID CPUS=$SLURM_CPUS_ON_NODE GPUS=$SLURM_GPUS_ON_NODE"
echo "NERSC_HOST=$NERSC_HOST"
sacctmgr -nP show assoc user="$USER" format=Account,QOS
If hostname matches login* and $SLURM_JOB_ID is empty, you're on a login node — fine for editing, git, module avail, light Python, small file ops. Do not run sustained compute, multi-core, or big-memory work there. Login nodes throttle CPU and are shared with hundreds of other users; jobs that misbehave can be killed without warning.
Choosing how to launch a job
| Situation | Use |
|---|
| Quick interactive work, debugging | salloc -q interactive (CPU) or -q gpu_interactive (GPU) |
| Short test runs (≤30 min), high priority | -q debug / -q gpu_debug |
| Long-running interactive (Jupyter, tmux) | JupyterHub — survives SSH disconnects |
| Batch / fire-and-forget | sbatch -q regular / -q gpu_regular |
| Many parameter variants | sbatch --array=1-N |
| Cheap, restartable, can be preempted | -q preempt / -q gpu_preempt |
| Out of allocation but need to finish | -q overrun (free, lowest priority) |
salloc patterns
salloc -N 1 -C cpu -q interactive -t 60 -A <repo>
salloc -N 1 -C gpu -q gpu_interactive -t 60 -A <repo>_g --gpus=4
salloc -N 1 -C cpu -q shared -c 16 --mem=32G -t 2:00:00 -A <repo>
sbatch boilerplate
#!/bin/bash
module load python
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MPICH_GPU_SUPPORT_ENABLED=1
srun -n $SLURM_NTASKS python myscript.py ...
A ready-to-edit copy is at templates/job.sbatch. For partition/QOS tradeoffs see references/qos.md; for GPU specifics see references/gpu.md.
Always launch with srun from inside a job
This is the NERSC-specific thing that bites people coming from other Slurm sites: a bare python foo.py inside an sbatch script runs on the head compute node only and ignores most of the resource request. srun -n $SLURM_NTASKS ... is what places tasks across the allocation, sets up GPU binding, and is the only thing that gives you the resource accounting you asked for. Even for single-task jobs, srun is the right habit — it's free and correct.
Filesystems — where to write what
| Path | Var | Purpose | Backup | Quota / Retention |
|---|
/global/homes/<l>/<user> | $HOME | Code, dotfiles, venvs. Slow metadata under load. | Snapshots + tape | 40 GB / 1M inodes |
/pscratch/sd/<l>/<user> | $PSCRATCH (alias $SCRATCH) | Big I/O, intermediate outputs, training data, sims. Lustre. | None | ~20 TB / 10M inodes; purged after inactivity (currently ~8 weeks; verify with myquota / NERSC docs) |
/global/cfs/cdirs/<project> | $CFS for the prefix | Project-shared data, durable. GPFS. | Snapshots vary by project | Per-project quota |
/global/common/software/<project> | — | Read-only shared software install. Fast for many small reads (env activation, import). | — | Read-only on compute |
Per-job $TMPDIR | $TMPDIR | Per-node temp; auto-cleaned on job exit. | — | RAM-backed; small |
Default outputs to $PSCRATCH unless the user specifies otherwise — that's where the I/O budget lives, and $HOME will run out of inodes quickly if you write nightly logs/checkpoints there. Move keepers to a CFS project dir before the purge window.
Quota check:
scripts/check_quota.sh
myquota
prjquota <project>
More detail: references/filesystems.md.
Modules and Python environments
NERSC uses Cray's module system (Lmod underneath). module avail, module load X. Don't pip install into a system Python — make a venv or conda env that lives on $HOME or (better, for fast import) on /global/common/software/<project>/....
module load python
mamba create -p /global/common/software/<project>/envs/<name> python=3.12
source activate /global/common/software/<project>/envs/<name>
python -m venv $HOME/.venvs/<name>
source $HOME/.venvs/<name>/bin/activate
Always check echo $PYTHONPATH after activation — a PYTHONPATH set in ~/.bashrc is prepended to every Python and silently overrides venv installs.
Common gotchas
These bite everyone at least once on Perlmutter:
-
--account is mandatory for Slurm jobs. No default. Without it, sbatch/salloc errors out. GPU jobs need the _g-suffixed repo (e.g. m1234_g), not the CPU one.
-
srun is required to actually use the allocation. Bare python inside sbatch runs on the head node only. Always wrap the work in srun.
-
Login nodes throttle CPU. Anything multi-minute / multi-core / >a few GB belongs in salloc/sbatch. Symptom of forgetting: a job that's "just slow" — it isn't slow, it's being throttled.
-
$PSCRATCH is purged. Check the policy on your account; assume ~8 weeks of inactivity. Touch keepers (touch -a) or move them to CFS.
-
$HOME inode limit (1M) is the silent killer, not space. Tools that drop millions of small files (pip caches, tarball extractions, conda) fill inodes long before bytes. Symptom: cryptic No space left on device with df -h showing free space.
-
module load on the login node ≠ on compute. Allocations get a fresh environment unless you propagate it (#SBATCH --export=ALL is the default, but be explicit if it matters). Best practice: module load inside the sbatch script.
-
Preempt and overrun jobs can restart mid-run. Use --open-mode=append and write idempotent code if you choose those QOSes.
-
CFS metadata operations are slow under load — avoid find or ls -R over million-file directories. Use Globus or tar for bulk moves.
-
GPU-aware MPI needs export MPICH_GPU_SUPPORT_ENABLED=1 in the job script. Without it, MPI_Send of GPU buffers will silently corrupt or crash.
-
Don't bypass safety hooks (git commit --no-verify, pip install --break-system-packages, etc.). Diagnose the root cause.
More edge cases: references/gotchas.md.
Surviving SSH disconnects
salloc interactive shells die when SSH closes. Two patterns that survive:
setsid nohup srun -n 1 python -u long_job.py > logs/job.log 2>&1 < /dev/null &
echo "PID=$!"
For long interactive sessions (Jupyter, an editor) use NERSC's JupyterHub — it spawns its own Slurm job and persists across browser disconnects.
Git and GitHub workflows at NERSC
module load gh
gh auth login
git config --global user.name "Your Name"
git config --global user.email "you@..."
ssh-keygen -t ed25519
ssh -T git@github.com
Patterns that work well on the cluster:
gh pr checkout <N> to test a teammate's branch in the real NERSC environment (matches production better than their laptop).- Long edit/commit loops via tmux on the login node, or via JupyterHub terminal.
- Confirm before push / force-push / history-rewrite — those need explicit user authorization.
See references/git-on-nersc.md for extended workflows.
Bundled scripts
Available under scripts/:
| Script | What it does |
|---|
where_am_i.sh | Login vs compute, $NERSC_HOST, allocation state, available repos. Run first. |
check_quota.sh | myquota plus CFS project quotas you have access to. Run before large writes. |
Run any of them directly: scripts/check_quota.sh. No dependencies beyond a NERSC shell.
When you're not sure
