running-python

name	running-python
description	Run Python on the Yale SOM HPC cluster with uv, Slurm, thread control, logging, and resumable outputs. TRIGGER when writing Python sbatch scripts for the Yale SOM HPC cluster, creating uv environments under /gpfs, or debugging Python Slurm jobs on the cluster.
related	["coding-in-python","programming-and-coding","code-review","installing-software","managing-jobs","parallel-python","accelerating-python","using-gpus","acquiring-data"]
updated	2026-06-10T00:00:00.000Z

Running Python

Rule: use a project environment, control threads, log clearly, and make outputs resumable.

What you get: with uv you control the interpreter and every package yourself, so nothing depends on what Python happens to be preinstalled on the cluster — no bootstrap step to worry about.

Tooling defaults

Use uv. It is the package manager for cluster Python work; everything below assumes it. Don't reach for conda, mamba, poetry, pipenv, pip-tools, or pip install --user — uv supersedes all of them. Why specifically uv on this cluster:

• Single tool for dependencies, lockfile, virtualenv, and Python interpreter — no module/conda/pyenv stack to coordinate. The cluster's system python3 is old, and the python modules are a fixed set that changes between maintenance windows (check python3 --version and module spider python); uv downloads and pins whatever recent Python your project needs (pyproject.toml's requires-python).
• Lockfile (uv.lock) is built-in and resolves identically on the login node, compute node, and your laptop — no conda env export games, no "works on my machine."
• 10–100× faster than conda on GPFS. A uv sync --frozen is a few seconds; a conda env create is a multi-minute metadata storm because conda writes thousands of small files into one directory.
• Single .venv directory in your project — easy to inspect, easy to nuke, easy to atomically swap.

These pair with uv:

• ruff for lint + format. Catches mistakes locally before burning a Slurm allocation.
• pyrefly for type checking. Same reason as ruff.
• pytest for tests. Smoke tests on small inputs save many cluster reruns.
• argparse for batch scripts (one-file entry points like run_task.py --task-id). click only when you grow into a reusable project CLI; the extra dependency is not worth it for a single sbatch script.
• pathlib over os.path. Joining paths and checking parents is what you do most on the cluster.
• logging as the baseline (configured below). loguru is fine when its structured output materially helps incident debugging.
• pyproject.toml + uv.lock committed; .venv/ gitignored. The lockfile is what makes runs reproducible across login and compute nodes.

How to install the tools themselves: see installing software for uv (one curl command into ~/.local/bin). Once uv is on your PATH, ruff, pyrefly, and pytest go in your project's dev dependencies via uv add --dev ruff pyrefly pytest, so they reproduce from uv.lock like everything else.

Project setup with uv

Pin a recent Python in pyproject.toml and let uv install it — don't depend on the cluster's system python3 (old enough that NumPy 2.x and many libraries are dropping support — check python3 --version) or a module load python/... (a fixed version that can change between maintenance windows; see module spider python):

cd /gpfs/project/myproject/code
uv init --app --python 3.13
uv add polars pyarrow duckdb
uv sync --frozen

uv python list shows what's available; uv python install 3.13 downloads it into ~/.local/share/uv/python/ (~50 MB per version) if uv hasn't already. The pinned version is recorded as requires-python in pyproject.toml, so anyone running uv sync on this project gets the same interpreter. Pick a current Python (uv python list shows options); drop one minor version if a critical dependency lags.

This is a setup-time operation, run once on a login node. Do not run uv sync inside Slurm jobs or job arrays — environment mutation in flight is a waste pattern (and --frozen makes it explicit that the lockfile is the source of truth).

Commit:

pyproject.toml
uv.lock

Do not commit .venv/; put it in .gitignore.

Don't pip install --user or run pip install inside jobs — neither is reproducible and both leak state between projects. For a one-off package, uv add <pkg> then uv sync --frozen. See installing software for the broader picture.

Data work, default picks

For most cluster work, these are the right defaults:

• Tabular reads/writes → Polars for new code (uses your CPU allocation through threading and lazy scan_*); pandas at API boundaries (sklearn, statsmodels, plotting). Convert with .to_pandas() only at the boundary; round-tripping doubles memory.
• File format → Parquet with compression="zstd". One reused Parquet beats 10k CSVs both for speed and for GPFS metadata health.
• Lazy reads → pl.scan_csv / pl.scan_parquet push filters and column projection before materialization, keeping memory under your --mem limit.
• Append-heavy / streaming output → JSONL with gzip is the simplest correct option for record-by-record writes (one append-only file, atomic at line granularity). For columnar appends, write one Parquet per task or per chunk (out/task_0001.parquet, out/task_0002.parquet, …) and read them back with pl.scan_parquet("out/*.parquet"). Do not mutate one big Parquet in place. See using the filesystem for the full append patterns.
• SQL over local files → DuckDB. Joins CSV/Parquet/JSON without staging.
• Per-project SQL state — catalogs, progress, lookups, OLTP-style writes → SQLite with WAL (PRAGMA journal_mode=WAL, synchronous=NORMAL, busy_timeout=15000). Handles concurrent writers (serialized via file locking) and non-blocking readers; fine on GPFS and on compute-node /local. The full pragma set + a batched ArtifactWriter pattern live in acquiring data.
• Multi-user database → psycopg with psycopg_pool; create one pool per process if you fork.
• Unknown encodings → charset-normalizer to detect, then pass encoding= explicitly.

Worked examples for query patterns and ingestion live in accelerating Python and acquiring data.

Safe Python Slurm template

Use this shape as the default. The launch line is srun .venv/bin/python ..., not uv run python ..., so SIGUSR1 reaches Python on long jobs (see parallel Python for why):

#!/bin/bash
#SBATCH --job-name=python-job
# default_queue caps at 4h; for long/large work use cpunormal or gpunormal (see managing-jobs)
#SBATCH --partition=default_queue
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --output=logs/%x_%j.out

set -euo pipefail

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
export MKL_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
export OPENBLAS_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
export NUMEXPR_NUM_THREADS=${SLURM_CPUS_PER_TASK:-1}
export PYTHONUNBUFFERED=1                              # Slurm logs update during the job, not just at exit

cd /gpfs/project/myproject/code
# Environment was created during setup with: uv sync --frozen
srun .venv/bin/python src/main.py

For short, exploratory one-off jobs where signal-based shutdown does not matter, uv run python src/main.py (without srun) is acceptable. For anything long-running or resumable, use the srun .venv/bin/python form.

Read Slurm settings safely

SLURM_* env vars are only set inside Slurm jobs. The patterns below let the same script run on your laptop (no Slurm) and on the cluster (Slurm fills in real values):

import os

# Slurm allocation when running under sbatch/srun, fall back to the
# laptop's CPU count for local testing.
n_cpus = int(os.environ.get("SLURM_CPUS_PER_TASK", "0")) or os.cpu_count() or 1

# "local" is a useful sentinel for log file names off-cluster.
job_id = os.environ.get("SLURM_JOB_ID", "local")

The first line reads as "use Slurm's CPU count if it's set, else os.cpu_count(), else 1." On the cluster you get the allocation; on a laptop you get the local CPU count; in a constrained container you still get a sensible non-zero number.

Logging

import logging
import os

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

logging.info("job_id=%s", os.environ.get("SLURM_JOB_ID", "local"))

Set PYTHONUNBUFFERED=1 in the Slurm script (above) so log lines reach logs/*.out while the job is running, not all at once at the end. Use logs to know what happened without opening notebooks.

Multiprocessing

For nontrivial multiprocessing, producer-consumer queues, process pools, or job-stealing-style work, use parallel Python.

Minimal rule: match worker count to allocated CPUs.

from concurrent.futures import ProcessPoolExecutor
import os

n_workers = int(os.environ.get("SLURM_CPUS_PER_TASK", "0")) or os.cpu_count() or 1

with ProcessPoolExecutor(max_workers=n_workers) as pool:
    results = list(pool.map(run_one_task, tasks))

Avoid nested parallelism: Slurm array × Python multiprocessing × BLAS threads can explode CPU usage.

Resumable numbered tasks

Write scripts so each Slurm array task can be restarted safely. If the output for task 17 exists, task 17 exits without doing work.

import argparse
from pathlib import Path

import polars as pl

parser = argparse.ArgumentParser()
parser.add_argument("--task-id", type=int, required=True)
args = parser.parse_args()

output = Path(f"/gpfs/project/myproject/output/task_{args.task_id:04d}.parquet")
if output.exists():
    print(f"task {args.task_id} already done: {output}", flush=True)
    raise SystemExit(0)

# Replace this with real task-specific work.
result = pl.DataFrame({"task_id": [args.task_id], "value": [args.task_id ** 2]})

tmp = output.with_suffix(".parquet.tmp")
result.write_parquet(tmp)
tmp.rename(output)

This pattern lets you rerun the same array and only compute missing tasks.

GPU check

Only use this in a GPU allocation:

import torch

assert torch.cuda.is_available(), "No CUDA device visible"
print(torch.cuda.get_device_name(0))

Checklist

• Project uses uv (not conda / poetry / pipenv / pip+venv); pyproject.toml and uv.lock are committed.
• pyproject.toml pins a recent Python via requires-python (e.g. >=3.13); the project does not depend on system python3 or module load python.
• uv sync --frozen runs at setup, never inside arrays or jobs.
• Slurm script sets OMP/MKL/OPENBLAS/NUMEXPR_NUM_THREADS and PYTHONUNBUFFERED=1.
• Long or resumable jobs launch with srun .venv/bin/python ..., not uv run python ..., so SIGUSR1 reaches Python.
• Python reads SLURM_CPUS_PER_TASK with a laptop-friendly fallback (os.cpu_count()).
• Multiprocessing workers do not exceed allocated CPUs.
• Outputs are skip-if-exists and atomically written.
• Logs include job ID and key progress messages.

Further reading

• uv documentation — projects, environments, lockfiles, uv run.
• ruff and pyrefly — lint/format and type checking.
• Polars user guide — eager and lazy DataFrames, scan_*, expressions.
• DuckDB Python API — SQL over local files.
• psycopg 3 and psycopg_pool — PostgreSQL with pooled connections.
• Slurm sbatch reference — directives and SLURM_* env vars Python reads.
• logging module — handlers, formatters, levels.
• argparse module — CLI args for resumable task scripts.