self-diagnosing-resource-use

name	self-diagnosing-resource-use
description	Diagnose whether a Yale SOM HPC cluster Slurm job used its requested CPUs, memory, GPUs, and time wisely, then right-size the next request. TRIGGER when a Slurm job on the Yale SOM HPC cluster is slow, killed, pending, idle, over-requested, ran out of memory, or failed, or whenever a job just finished and should be checked for waste.
related	["managing-jobs","using-gpus","using-the-filesystem","running-python","accelerating-python"]
updated	2026-06-09T00:00:00.000Z

Self-Diagnosing Resource Use

Rule: after every serious job, check what you actually used and right-size the next job. Do this on your own initiative and report the result in plain language — the user is usually a researcher who will not think to ask "was that wasteful?" Translate the numbers ("used 6 of 64 GB; drop --mem to 12G next time"), don't just paste seff output.

Completed job accounting

sacct -j JOBID --format=JobID,JobName,Elapsed,AllocCPUS,TotalCPU,MaxRSS,State --units=G

Note: MaxRSS is blank on the parent job row — it only populates on the .batch and step (.0, .1, …) rows. Read the .batch line for actual peak memory, not the summary line.

For a friendlier summary:

seff JOBID

Caveat: seff samples coarsely and reports CPU Utilized: 00:00:00 / 0.00% for sub-minute jobs even when they did real work (verified on this cluster). Ignore the CPU-efficiency numbers on jobs shorter than ~1–2 minutes — they reflect the sampling miss, not waste. Trust them only on jobs long enough to be sampled.

Interpret CPU use

Approximate CPU efficiency:

TotalCPU / (Elapsed × AllocCPUS)

Rules of thumb:

• 50%: reasonable for CPU-bound jobs.

• 10–50%: maybe I/O-bound or over-requested.
• <10%: probably wasteful; request fewer CPUs or parallelize correctly.

Interpret memory use

If MaxRSS is 4 GB and you requested 128 GB, lower --mem next time. Aim for ~1.5–2× the observed peak, not 10×.

Why this matters: Slurm reserves the full requested memory whether or not your job uses it. A user requesting 1 TB for a 10 GB job is removing 1 TB from everyone else's available pool until the job ends. There are no per-user caps enforcing this — it runs on courtesy — so right-sizing is simply good citizenship on a shared machine.

Check current jobs

squeue -u $USER -o "%.8i %.9P %.20j %.2t %.10M %.6D %.4C %.10m %R"

GPU diagnosis

Inside a GPU allocation:

watch -n 1 nvidia-smi

Log for later:

nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,memory.total,power.draw \
  --format=csv -l 10 > logs/gpu_${SLURM_JOB_ID}.csv &

Rules of thumb:

• 0 MB used: your process is not using the GPU.
• Low GPU utilization and high VRAM: model loaded but waiting on data/CPU/network.
• Sustained <10% GPU utilization: cancel and diagnose.

Filesystem impact

du -sh /gpfs/scratch60/$USER 2>/dev/null
find /gpfs/scratch60/$USER -mtime +30 -size +1G -ls 2>/dev/null | head
find output -type f | wc -l

If a job creates thousands of files, strongly consider redesigning the output storage. See using the filesystem for Parquet, JSONL, zip, local /tmp, and atomic-write patterns.

Agent-friendly checkup script

#!/bin/bash
set -euo pipefail

jobid=${1:?usage: ./checkup.sh JOBID}

echo "=== Job accounting ==="
sacct -j "$jobid" --format=JobID,JobName,Elapsed,AllocCPUS,TotalCPU,MaxRSS,State

echo ""
echo "=== seff ==="
seff "$jobid" 2>/dev/null || true

echo ""
echo "=== Current jobs ==="
squeue -u "$USER" -o "%.8i %.9P %.20j %.2t %.10M %.6D %.4C %.10m %R"

echo ""
echo "=== Scratch usage ==="
du -sh "/gpfs/scratch60/$USER" 2>/dev/null || true
find "/gpfs/scratch60/$USER" -mtime +30 -size +1G -ls 2>/dev/null | head || true

echo ""
echo "=== GPU status, if on GPU node ==="
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv 2>/dev/null || true

Quick rules table

Metric	Good	Wasteful	Fix
CPU efficiency	>50%	<10%	request fewer CPUs or parallelize
Memory use	>25% requested	<5%	reduce --mem
GPU utilization	>30% training	<10%	split preprocessing, fix data loader, or use CPU
GPU memory	meaningful fraction	0 MB	code is not on GPU
File count	<1,000/job	>10,000/job	Parquet/HDF5/zip
Time used	close to request	tiny fraction	request shorter time

Checklist

• Checked sacct or seff after the job.
• Reduced future CPU/memory/time requests if over-requested.
• Checked nvidia-smi for GPU jobs.
• Checked scratch usage and old files.
• Confirmed output file count is reasonable.
• Made the next job shorter or more resumable where possible.

Further reading

• Slurm sacct — format strings, fields like MaxRSS, TotalCPU, Elapsed.
• Slurm squeue — format strings and reason codes for pending jobs.
• nvidia-smi reference — --query-gpu, logging utilization, MIG.
• py-spy — py-spy dump --pid PID for stuck Python processes.