| name | dunnlab-hpc |
| description | YCRC HPC cluster reference for the Dunn Lab. Use when writing SLURM batch scripts, configuring job resources, managing storage, or running analyses on Bouchet, McCleary, or Misha clusters. Covers partitions, storage, job scheduling, and cluster-specific details.
|
YCRC HPC Clusters
The Dunn Lab runs computationally intensive analyses on Yale Center for Research Computing (YCRC) clusters. Full documentation: https://docs.ycrc.yale.edu/
Bouchet is the lab's primary cluster. McCleary is retained only for YCGA raw sequence data work.
Cluster overview
| Cluster | Focus | Status | SSH | OOD Portal |
|---|
| Bouchet | General HPC (successor to Grace & McCleary) | Active — primary cluster | bouchet.ycrc.yale.edu | ood-bouchet.ycrc.yale.edu |
| McCleary | YCGA sequencing & CryoEM | YCGA-only — non-YCGA work has moved to Bouchet | mccleary.ycrc.yale.edu | ood-mccleary.ycrc.yale.edu |
| Misha | Wu Tsai Institute (neuroscience & data science) | Active | misha.ycrc.yale.edu | ood-misha.ycrc.yale.edu |
| Hopper | Regulated/sensitive data (incl. NIH Controlled Access) | Active | — | — |
| Grace | Former general HPC | Retired — decommissioned in the 2026 migration | — | — |
Request accounts at https://research.computing.yale.edu/account-request. New McCleary accounts are only approved for groups using YCGA resources, CryoEM resources, or dedicated nodes.
Transfer nodes: transfer-bouchet.ycrc.yale.edu, transfer-mccleary.ycrc.yale.edu. For large or cross-cluster transfers, YCRC recommends Globus.
Login node policy
Never run heavy computation on login nodes. The following lightweight tasks are acceptable on login nodes:
- Snakemake orchestration (dispatching jobs to SLURM)
- Git operations
- Conda environment management
- File inspection, editing scripts
- Small interactive commands (
wc, head, ls, etc.)
Everything else must be submitted as a SLURM job.
Bouchet
Docs: https://docs.ycrc.yale.edu/clusters/bouchet/ · Getting started: https://docs.ycrc.yale.edu/clusters/bouchet_getting_started/
Bouchet is hosted at MGHPCC and is where all HPC growth and refreshes are deployed. Compute is a mix of Intel Emerald Rapids (Xeon 8562Y, 64 cores / 990 GiB) and AMD Turin (EPYC 9575F, 128 cores / 2,251 GiB; EPYC 9655, 192 cores / 1,487 GiB) nodes. GPUs span RTX 5000 Ada, A40, L40S, H200, B200, and RTX PRO 6000 Blackwell.
Bouchet account and path conventions
Bouchet differs from the older clusters in ways that break copied scripts:
- Your primary group is your NetID; PI groups are
pi_<netid> (not the PI's last name). Check with groups and slurm_checkup.sh.
- Storage lives under
/nfs/roberts/ on the all-flash Roberts filesystem. There is no GPFS and no /vast/palmer — paths like /gpfs/gibbs/project and ~/palmer_scratch do not exist here. Disable any GPFS-specific optimizations in tools.
- Home symlinks are
~/project_pi_<netid> and ~/scratch_pi_<netid> (not ~/project / ~/palmer_scratch).
- Software is built against the 2022b and 2024a toolchains only.
- Conda environments cannot be copied from Grace/McCleary — rebuild them, or migrate with
conda-pack.
Run mydirectories to print your actual paths and getquota to check usage.
Bouchet partitions
Public partitions (private priority*, pi_*, and education* partitions are excluded). Verify against live state with sinfo.
| Partition | Max Time | Nodes | CPUs/Node | RAM/Node | Per-user limits |
|---|
| day | 1 day | ~110 | 64 / 128 / 192 | 990 / 2,251 / 1,487 GiB | 1,000 CPUs, 15 TiB mem — default partition |
| devel | 6 hours | 5 | 64 / 192 | 990 / 1,487 GiB | 4 CPUs, 60 GiB, 2 jobs |
| week | 7 days | 14 | 128 | 2,251 GiB | 96 CPUs, 1.5 TiB mem |
| bigmem | 1 day | 4 | 64 | 4,014 GiB | 128 CPUs, 8 TiB mem |
| mpi | 2 days | 60 | 64 | 487 GiB | 48 nodes, 10 jobs |
| gpu | 2 days | 29 | 48 | 479–977 GiB | 16 GPUs, 12 jobs |
| gpu_h200 | 2 days | 9 | 48 | 1,996 GiB | 16 GPUs, 6 jobs |
| gpu_b200 | 2 days | 7 | 128 | 2,251 GiB | 16 GPUs |
| gpu_rtx6000 | 2 days | 7 | 128 | 2,251 GiB | 16 GPUs |
| gpu_devel | 6 hours | 6 | 48 / 128 | 479–2,251 GiB | 12 CPUs, 2 GPUs, 1 job |
| scavenge | 1 day | ~230 | 32–192 | 488–4,014 GiB | Preemptable; idle private nodes |
| scavenge_gpu | 1 day | ~48 | 32–128 | 488–2,251 GiB | Preemptable GPU nodes |
Bouchet GPUs
Request GPUs by type, not just count, so you land on hardware that fits the job:
| GPU type | SLURM name | vRAM | Per node | Partition |
|---|
| NVIDIA RTX 5000 Ada | rtx_5000_ada | 32 GB | 4 | gpu, gpu_devel |
| NVIDIA A40 | a40 | 48 GB | 4 | gpu |
| NVIDIA L40S | l40s | 48 GB | 4 | gpu |
| NVIDIA H200 | h200 | 141 GB | 8 | gpu_h200, gpu_devel |
| NVIDIA RTX PRO 6000 Blackwell | rtx_pro_6000_blackwell | 96 GB | 8 | gpu_rtx6000, gpu_devel |
| NVIDIA B200 | b200 | 180 GB | 8 | gpu_b200, gpu_devel |
Example: #SBATCH --gpus=h200:1. List current inventory with sinfo -e -o "%20P %6D %60G".
Bouchet storage
| Name | Path | Home symlink | Quota | File limit | Backed Up | Purge Policy |
|---|
| Home | /nfs/roberts/home/<netid> | ~/ | 125 GiB/user | 500K | Yes (snapshots >= 2 days) | None |
| Project | /nfs/roberts/project/pi_<netid> | ~/project_pi_<netid> | 4 TiB/group | 5M | Yes (snapshots >= 2 days) | None |
| Scratch | /nfs/roberts/scratch/pi_<netid> | ~/scratch_pi_<netid> | 10 TiB/group | 15M | No | 60-day purge |
| PI | /nfs/roberts/pi/<grp> | — | Purchased | — | Snapshots >= 2 days | None |
No sensitive or regulated data may be stored on Bouchet — use Hopper.
McCleary
Docs: https://docs.ycrc.yale.edu/clusters/mccleary/ · Decommission plan: https://docs.ycrc.yale.edu/clusters/grace-mccleary-decommission/
McCleary is winding down to a YCGA-only cluster. The lab retains access for YCGA raw sequence data work; run everything else on Bouchet.
Migration status:
- Phase 1 (complete) — groups without dedicated nodes, CryoEM, or YCGA affiliation lost Grace/McCleary access on June 1, 2026.
- Phase 2 (late 2026 / early 2027) — non-YCGA workloads and data belonging to YCGA-affiliated groups move to Bouchet. Anything you want to keep off McCleary must be transferred by then.
- Phase 3 (late 2026 / early 2027) — newer commons and dedicated nodes relocate to Bouchet; Grace shuts down. McCleary persists as YCGA-only for the remaining life of YCGA-owned hardware.
NIH Controlled Access Data cannot move to Bouchet — it must go to Hopper.
Hardware: heterogeneous Intel Xeon (Skylake, Cascade Lake, Ice Lake), 163–3,960 GiB per node. GPUs include A100 (40/80 GB), V100, A5000, RTX 3090, RTX 5000, L40S.
McCleary partitions
| Partition | Max Time | Max CPUs/User | Max Mem/User | Max GPUs/User |
|---|
| day | 1 day | 256 | 3,000 GiB | — |
| devel | 6 hours | 4 | 32 GiB | — |
| week | 7 days | 192 | 2,949 GiB | — |
| long | 28 days | 36 | — | — |
| gpu | 2 days | — | — | 12 |
| gpu_devel | 6 hours | 10 | — | 2 |
| bigmem | 1 day | 32 | 3,960 GiB | — |
| scavenge | 1 day | 1,000 | 20,000 GiB | — |
| scavenge_gpu | 1 day | — | — | 64 |
| ycga | — | — | — | Submit YCGA work here to avoid CPU charges |
McCleary storage
| Name | Path | Quota | Backed Up | Purge Policy |
|---|
| Home | ~/ | 125 GiB, 500K files | Yes (backed up + snapshotted) | None |
| Project | ~/project (/gpfs/gibbs/project) | 4 TiB/group, 5M files | Snapshotted | None |
| Scratch | ~/palmer_scratch (/vast/palmer/scratch) | 10 TiB/group, 15M files | No | 60-day purge |
| PI | /gpfs/gibbs/pi/<grp> or /vast/palmer/pi/<grp> | Purchased | Varies | None |
Check quotas: getquota | List paths: mydirectories
McCleary defaults
Unless overridden: 1 hour walltime, 1 node, 1 task, 1 CPU, 5120 MB memory per CPU.
Misha
Docs: https://docs.ycrc.yale.edu/clusters/misha/
Hardware: Intel Sapphire Rapids and Emerald Rapids (6458, 6542, 6442, 6326). Standard nodes: 64 CPUs, 479 GiB. GPUs: H100 (80 GB), H200 (141 GB), A100 (80 GB), A40 (48 GB), L40S (48 GB).
Misha partitions
| Partition | Max Time | Nodes | CPUs/Node | RAM/Node | Per-User Limits |
|---|
| day | 1 day | 18 | 64 | 479 GiB | 512 CPUs, 20 TiB mem |
| devel | 6 hours | 2 | 64 | 479 GiB | 10 CPUs, 70 GiB |
| week | 7 days | 6 | 64 | 479 GiB | 128 CPUs, 1,280 GiB |
| gpu | 2 days | 31 | 32-48 | 975-1000 GiB | 192 CPUs, 18 GPUs |
| gpu_devel | 6 hours | 2 | 32 | 975 GiB | 4 CPUs, 1 GPU |
| bigmem | 1 day | 2 | 64 | 1,991 GiB | 64 CPUs, 2 TiB mem |
Misha storage
| Name | Path | Quota | Backed Up | Purge Policy |
|---|
| Home | /gpfs/radev/home | 125 GiB, 500K files | Snapshots >= 2 days | None |
| Project | /gpfs/radev/project | 1 TiB/group, 5M files | Snapshots >= 2 days | None |
| Scratch | /gpfs/radev/scratch | 10 TiB/group, 15M files | No | 60-day purge |
SLURM job scheduling
Full docs: https://docs.ycrc.yale.edu/clusters-at-yale/job-scheduling/
Batch script template
#!/bin/bash
module reset
module load miniconda
conda activate myenv
<command>
Critical: No space between # and SBATCH — otherwise the directive is ignored.
Use module reset, not module purge. module reset restores the sticky StdEnv module, which sets CONDA_ENVS_PATH, SLURM_HINT=nomultithread, the default salloc partition, and the /apps/bin PATH entry. module purge strips StdEnv and leaves a broken environment.
batch.sh conventions
Every computationally intensive script or pipeline should include a companion batch.sh SLURM submission script in the same directory. This makes it clear how to run the code and with what resources.
- Name the file
batch.sh and place it alongside the script it runs. If there are multiple stages, use separate scripts (batch_align.sh, batch_assemble.sh, etc.).
- Set
--job-name to something descriptive (e.g., the analysis name or script name).
- Direct SLURM output to
logs/ (e.g., --output=logs/slurm-%j.out) — ensure the directory exists before submission.
- Include
module reset before loading modules to avoid environment conflicts.
- Size resource requests to match actual needs — check with
jobstats after initial runs and adjust.
- For array jobs processing many samples, use
--array and document the expected input format.
- Document how to launch jobs in the project README. At minimum, include the submission command (e.g.,
sbatch batch.sh) and any prerequisites such as creating the logs/ directory or activating a conda environment beforehand.
Common directives
| Directive | Short | Default | Purpose |
|---|
--job-name | -J | — | Job identification |
--time | -t | 1 hour | Walltime (D-HH:MM:SS) |
--partition | -p | day | Target partition |
--nodes | -N | 1 | Compute nodes |
--ntasks | -n | 1 | MPI task count |
--cpus-per-task | -c | 1 | Cores per task (for threading) |
--mem-per-cpu | — | 5 GiB | RAM per CPU |
--gpus | -G | 0 | GPU count (--gpus=<type>:<n>) |
--output | -o | — | Output file (%j = job ID) |
--mail-type | — | — | Email notifications (ALL, FAIL, etc.) |
GPU jobs
#!/bin/bash
module reset
module load CUDA miniconda
conda activate myenv
python train.py
GPUs must be explicitly requested with --gpus; none are allocated by default. On Bouchet, match the partition to the GPU type — see the Bouchet GPU table above.
Interactive jobs
salloc -p devel -t 2:00:00 --mem=8G
StdEnv sets SALLOC_PARTITION=devel, so salloc targets devel unless you pass -p. Add --x11 for graphical forwarding (requires X11 setup). Wrap interactive sessions in tmux so a dropped connection does not kill the job.
Job monitoring
| Command | Purpose |
|---|
squeue --me | List your running/pending jobs |
sacct -j <id> | Job status and resource usage |
jobstats <id> | Efficiency metrics (CPU/memory utilization) |
scancel <id> | Cancel a job |
sbatch --test-only script.sh | Estimate queue start time without submitting |
slurm_checkup.sh | Show your SLURM accounts and group membership |
Job arrays and bulk submission
For many similar jobs, use job arrays (--array) or Dead Simple Queue rather than looping over sbatch. YCRC enforces job submission rate limits, and a submission loop will trip them.
module load dSQ
dsq --job-file joblist.txt --mem-per-cpu 4g -t 2:00:00 --submit
dsqa -j <arrayjobid>
Resource efficiency
Always check resource usage with jobstats after jobs complete. Request only what you need — wasteful allocations slow scheduling for everyone. Match --cpus-per-task to the actual thread count your tool uses. Match --mem-per-cpu to observed memory usage.
Snakemake + SLURM integration
Snakemake with --executor slurm (via snakemake-executor-plugin-slurm) submits each rule as an independent SLURM job. Snakemake itself runs on the login node as a lightweight orchestrator — this is acceptable.
Setup
module load miniconda
conda create -n snakemake -c conda-forge -c bioconda snakemake snakemake-executor-plugin-slurm
conda activate snakemake
Running
snakemake -n --executor slurm
snakemake --executor slurm --jobs 50
Per-rule resources in Snakefile
rule align:
resources:
slurm_partition="day",
runtime=240,
mem_mb=20000,
cpus_per_task=4,
slurm_extra="'--mail-type=FAIL'"
For GPU rules, add: slurm_partition="gpu", slurm_extra="'--gpus=rtx_5000_ada:1'"
Snakemake profile (recommended)
Create ~/.config/snakemake/slurm/config.yaml:
executor: slurm
jobs: 50
default-resources:
slurm_partition: day
runtime: 60
mem_mb: 5000
cpus_per_task: 1
latency-wait: 120
Then run: snakemake --profile slurm
Use tmux for long-running orchestration
tmux new -s pipeline
module load miniconda
conda activate snakemake
snakemake --executor slurm --jobs 50
Conda environment management
Prefer conda/mamba over pip. Always use conda-forge as the primary channel. Add bioconda when bioinformatics tools are needed.
module load miniconda
conda create -n myenv -c conda-forge python=3.11 <packages>
conda env create -f environment.yml
conda env export > environment.yml
conda install -c conda-forge <pkg>
pip install <pkg>
Store environments in your project directory or home — never in scratch (they will be purged after 60 days). Environments are not portable between clusters: rebuild on Bouchet, or migrate with conda-pack.
Important policies
- Scratch purge: Files older than 60 days on scratch are automatically deleted. You will receive email notification one week before deletion. Do not artificially extend file modification times to circumvent the policy.
- Sensitive data: No sensitive or regulated data on any cluster except Hopper. NIH Controlled Access Data must go to Hopper, not Bouchet.
- Max interactive apps: 4 concurrent OOD interactive instances per user.
- Job rate limits: YCRC enforces submission rate limits — use job arrays or
dsq instead of submission loops.
- Module system: Use
module reset then module load for software. Run module avail to see available packages.
- AI coding agents: YCRC does not formally support coding agents on the clusters and warns about data exposure, credential leakage, and destructive actions taken with your permissions. See https://docs.ycrc.yale.edu/ai/aicodingtools/ and use restrictive Claude Code permissions (
assets/settings.json in this repo).
- Paid storage: YCRC cannot accept new or increased paid storage allocations on Bouchet, Grace, or McCleary; availability may return in late 2026.
