engaging

name	engaging
description	MIT Engaging cluster reference. SLURM job scheduling, Coley group nodes, filesystem layout, Python environments, and common commands. Use when working with the Engaging HPC cluster.

MIT Engaging Cluster Reference

SLURM-managed HPC cluster run by MIT ORCD. Kerberos + Duo 2FA required (reducible via ORCD OnDemand or SSH control channels).

Tips adapted from the Coley Group Engaging Guide by Sam Goldman, David Graff, Runzhong Wang, and Magdalena Lederbauer.

Access

Login node (Rocky 8):

ssh [e-mail]

Two-Factor Authentication (Kerberos password + Duo) required. Reduce repeated 2FA by:

• Logging into ORCD OnDemand (temporarily allows SSH key login without password/Duo)
• Using SSH Control Channels (Mac/Linux) to multiplex connections

SSH config for convenience:

Host engaging
  Hostname orcd-login.mit.edu
  User <kerberos>
  ControlMaster auto
  ControlPath ~/.ssh/control-%r@%h:%p
  ControlPersist 8h

Coley Group Nodes

Partition	Node	CPUs	GPUs	GPU VRAM	RAM
sched_mit_ccoley	node1236	52 (2x Intel 6230R)	8x 2080 Ti	11 GB	514 GB
sched_mit_ccoley	node1238	128 (2x AMD EPYC 7702)	—	—	515 GB
pi_ccoley	node2436	64 (2x Intel 8562Y+)	4x H100	80 GB	1000 GB
pi_ccoley	node2519-2522	64 (2x AMD EPYC 7513)	—	—	249 GB

Resource etiquette for node1236: 52 CPUs and 514 GB across 8 GPUs. Use -n 6 --mem-per-cpu=8G per single-GPU job so all 8 GPUs can run simultaneously. Requesting too many CPUs or too much memory per job blocks other GPUs from being used.

Prefer CPU-only nodes (node1238, node2519-2522) for non-GPU work.

ChemE Department Nodes

Partition	Node	CPUs	GPUs	GPU VRAM	RAM	Notes
ou_cheme	node4110-4113, 4115-4116	128 (2x64 cores)	—	—	770 GB	also in mit_preemptable
ou_cheme	node4114, 4117	128 (2x64 cores)	—	—	1540 GB	high-memory, also in mit_preemptable
ou_cheme_gpu	node5000-5001	120 (2x60 cores)	8x H200	141 GB	2055 GB	also in mit_preemptable
ou_cheme_gpu	node5003-5005	128 (2x64 cores)	8x RTX PRO 6000	96 GB	2055 GB	also in mit_preemptable

All ou_cheme CPU nodes are in both ou_cheme and ou_cheme_preemptable partitions, and also in mit_preemptable. GPU nodes are in ou_cheme_gpu and mit_preemptable.

Public Partitions

Partition	Purpose	GPUs	Max Time
mit_normal	CPU batch/interactive	—	12 hrs
mit_normal_gpu	GPU jobs	L40S (44 GB), H100 (80 GB), H200 (140 GB)	6 hrs
mit_quicktest	Quick testing	—	15 min
mit_preemptable	Low-priority, may be preempted	A100/L40S/H100/H200	48 hrs

Request GPUs with -G [type]:[count] (e.g., -G l40s:1, -G h200:2). Default on mit_normal_gpu is L40S.

CPUs per GPU on mit_normal_gpu: L40S = 16, H200 = 15. Requesting more CPUs than this wastes a GPU slot.

Filesystem

Path	Size	Backed Up	Purpose
/home/<user>	200 GB	Yes (snapshots)	Code, config, small files
/home/<user>/orcd/pool	1 TB	No	Larger datasets (not I/O intensive)
/home/<user>/orcd/scratch	1 TB	No	I/O-heavy job data (flash storage)
/nfs/ccoleylab001	10 TB shared	No	Group shared data — don't be greedy
/nobackup	cluster-global	No	Semi-persistent intermediate files (periodically wiped)
/tmp	node-local	No	Fastest I/O, but not shared between nodes; gone after job ends

Storage hierarchy for jobs: Use /tmp or scratch for active computation. Stage data from pool. Keep code and configs in home.

Scratch files deleted after 6 months of inactivity. /nobackup wiped ~monthly.

SLURM Quick Reference

# check partition availability
sinfo -p sched_mit_ccoley
sinfo -p mit_normal_gpu -O Partition,Nodes,CPUs,Memory,Gres

# interactive session (1 GPU on group node)
salloc -p pi_ccoley --gres=gpu:1 -n 6 --mem-per-cpu=8G --time=3:00:00

# interactive session (public H100)
salloc -p mit_normal_gpu -G h100:1 -c 4 --mem=32G --time=6:00:00

# batch job
sbatch myjob.sh

# check your jobs
squeue --me

# cancel a job
scancel <jobid>

# job history and memory usage
sacct -j <jobid> -o JobID,JobName,State,ReqMem,MaxRSS,Elapsed --units=G

# check node resources
scontrol show node node1236

# who's using our partition
squeue -p sched_mit_ccoley

Example Batch Script

#!/bin/bash
#SBATCH -p pi_ccoley
#SBATCH --gres=gpu:1
#SBATCH -n 6
#SBATCH --mem-per-cpu=8G
#SBATCH -t 0-08:00:00
#SBATCH -J my_job
#SBATCH --output=logs/%j_%x.log

module load miniforge
source activate myenv

python train.py

Batch with Variable Command

#!/bin/bash
#SBATCH -p pi_ccoley
#SBATCH -n 6
#SBATCH --mem-per-cpu=8G
#SBATCH -t 0-05:00:00
#SBATCH --output=logs/%j_%x.log

module load miniforge
source activate myenv
eval $CMD

Launch with: sbatch --export=CMD="python train.py --lr 1e-4" generic_slurm.sh

Preemptable Jobs

Use --requeue flag so jobs restart when preempted. Ensure your code checkpoints regularly.

sbatch --requeue -p mit_preemptable -G l40s:1 myjob.sh

Python Environment Setup

# load system miniforge (do NOT install your own conda/miniconda)
module load miniforge

# option 1: conda/mamba environment
mamba create -n myenv python=3.12 pytorch torchvision
source activate myenv
# install pip-only packages after activating:
pip install some_package   # do NOT use --user flag

# option 2: virtual environment
python -m venv ~/myproject/venv
source ~/myproject/venv/bin/activate
pip install -r requirements.txt   # do NOT use --user flag

In job scripts, always load miniforge and activate the environment explicitly — don't rely on login shell state:

module load miniforge
source activate myenv

Things to Avoid

• Do NOT run conda init or mamba init — it modifies .bashrc and causes issues
• Do NOT install your own miniconda/miniforge/anaconda
• Do NOT use pip install --user — installs to ~/.local instead of the environment
• Do NOT set PYTHONPATH unless absolutely necessary

Conda Environments Filling Home?

Move them to scratch by creating ~/.condarc:

envs_dirs:
  - /home/USERNAME/orcd/scratch/.conda/envs
pkgs_dirs:
  - /home/USERNAME/orcd/scratch/.conda/pkgs
auto_activate_base: false

Back up environments with conda export --name myenv --file=environment.yaml.

Jupyter with GPU

Port-forwarding approach (better GPU than OOD's default K20):

# terminal 1: get a GPU allocation
ssh orcd-login.mit.edu
salloc -p sched_mit_ccoley -w node1236 --gres=gpu:1 -n 6 --mem-per-cpu=8G --time=3:00:00
# note the allocated node, e.g., node1236

# terminal 2: port forward through login node to compute node
ssh orcd-login.mit.edu -L 9998:localhost:9998
ssh node1236 -L 9998:localhost:9998
jupyter lab --port 9998

Copying Files

# from local machine to engaging
scp -r localdir/ [e-mail]:~/project/

# or with rsync (better for repeated syncs)
rsync -az --exclude '.venv*' --exclude '__pycache__' --exclude '.git' \
  . [e-mail]:~/project/

Troubleshooting

• Python can't find packages in job: Check which python points to your env. Load miniforge + activate env in the job script, not just at login.
• Conda activate fails in batch script: Add eval "$(conda shell.bash hook)" before conda activate.
• Home directory full: Check conda envs (du -sh ~/.conda), move to scratch (see above). Remove unused envs with conda remove -n old_env --all.
• Job pending (Resources): Requested resources not available yet. Check sinfo for node states.
• Job pending (Priority): Someone else is ahead in queue. Consider mit_preemptable for longer jobs.
• Wrong python version: which python to verify. Virtual envs inherit the python used to create them.