lumi-supercomputer

name	lumi-supercomputer
description	Use when running workloads on LUMI supercomputer, including GPU job submission, PyTorch with ROCm/AMD MI250X, container workflows, and LUMI-specific Slurm configuration. Triggers: "LUMI", "MI250X", "ROCm", "AMD GPU", "CSC", "LUMI-G"

LUMI Supercomputer

When to Use

• Submitting GPU jobs to LUMI (partitions, billing, resource requests)
• Setting up PyTorch on AMD MI250X GPUs (ROCm, not CUDA)
• Debugging LUMI-specific failures (MIOpen cache, Slingshot network, container issues)
• Planning multi-node distributed training on LUMI-G
• Configuring storage paths and understanding Lustre constraints

Quick Reference

Fact	Value
GPU	AMD Instinct MI250X (2 GCDs each = 8 logical GPUs/node)
GPU memory	64 GB HBM2e per GCD, 512 GB total/node
CPU	AMD EPYC 7A53 "Trento", 64 cores (56 usable)
CPU memory	512 GB DDR4 per node
Network	HPE Slingshot-11, 200 Gbps/NIC, 4 NICs/node
Software stack	ROCm (not CUDA), Singularity containers, module load PyTorch/...
ROCm default	6.3.4 (since Jan 2026 maintenance). PE versions: 25.03, 25.09
SSH	lumi.csc.fi
Outbound IP	193.167.209.128/26 (for firewall allowlists)
Quota check	lumi-workspaces (no module needed)
Allocation check	lumi-allocations
Docs	https://docs.lumi-supercomputer.eu
Support	https://lumi-supercomputer.eu/user-support

Key Differences from NVIDIA Clusters

These are the critical gotchas when moving from NVIDIA (Snellius, etc.) to LUMI:

1. ROCm, not CUDA: All GPU code runs through ROCm/HIP. PyTorch works transparently but CUDA-specific extensions (cuDNN calls, custom CUDA kernels) won't compile. Use ROCR_VISIBLE_DEVICES instead of CUDA_VISIBLE_DEVICES.

2. GCDs, not GPUs: Each MI250X has 2 Graphics Compute Dies. Slurm sees 8 "GPUs" per node, but they're 4 physical modules with 2 GCDs each. In-package bandwidth (same MI250X) is 4x faster than cross-module.

3. 56 cores, not 64: Low-noise mode disables 1 core per L3 region (8 regions) = 8 disabled. Only 56 are schedulable. Use --cpus-per-task=7 per GCD (not 8).

4. Container-first Python: Never pip install directly on Lustre. Use the provided Singularity containers via module load PyTorch/.... Extend with pip install inside the container shell, then make-squashfs to consolidate.

5. MIOpen cache on /tmp: MIOpen (AMD's cuDNN equivalent) uses a file-locked database. On Lustre this causes hangs. Always redirect: MIOPEN_USER_DB_PATH=/tmp/${USER}-miopen-cache.

6. Slingshot RCCL vars: Multi-GPU/multi-node communication requires explicit network config:

   export NCCL_SOCKET_IFNAME=hsn0,hsn1,hsn2,hsn3
   export NCCL_NET_GDR_LEVEL=3
   

7. No email notifications: Slurm on LUMI does not support --mail-type.

8. Auto-requeue: Enabled by default. Always use --no-requeue and --open-mode=append to avoid duplicated output.

9. Account is mandatory: Every job needs --account=project_<id>. Check allocation with lumi-allocations.

10. Jan 2026 maintenance broke old software: The system default ROCm is now 6.3.4 and PE versions are 25.03/25.09. Software installed before Jan 2026 may not run — recompilation or rebuilding with 25.03+ is often required. Older PE versions are unsupported.

11. Reload CrayEnv in job scripts: Login node environment is copied into jobs (with Rome/zen2 CPU targets). Reload CrayEnv in your sbatch script to pick up correct zen3 targets for LUMI-G compute nodes:

    module load CrayEnv  # resets CPU/network/accelerator targets for current node
    

12. zen3 CPU optimization: LUMI-G CPUs are AMD EPYC Trento (zen3 architecture). Compilers can optimize specifically for zen3 — the CrayEnv module auto-loads the correct target.

Launch Discipline

dev-g vs small-g Decision

Scenario	Partition	Walltime
Short single-GPU sanity, smoke test, latent extraction probe	dev-g	≤3 hrs
Resumable training chunk (checkpoint every hour)	dev-g	≤3 hrs
Longer production run, multi-GPU training	small-g	Up to 3 days

Never silently fall back from dev-g to small-g. If dev-g is unavailable, report it and wait or requeue explicitly.

Probe → Production Dependency Chain

Before any small-g production job:

1. Submit the same launcher/config as a dev-g probe with reduced steps/eval
2. Use 10-minute walltime for probes unless user specifies otherwise
3. Submit production with --dependency=afterok:<dev-g-job-id>

Concurrency limit: At most two dev-g probes active at once. When preparing more than two production jobs, chain probes with Slurm dependencies so the wave respects the limit.

CPU Jobs for Dataset Preparation

For raw dataset download, unpacking, or non-GPU preprocessing:

• Use CPU partitions (debug or small)
• Do not request GPUs
• Submit a short CPU probe first (one shard or dry-run that exercises auth, imports, storage writes, extraction)
• Submit full extraction only after probe succeeds

Reserve GPU jobs for GPU-bound preprocessing such as VAE latent extraction.

SSH Jump-Node Verification

Before any remote check, sync, preflight, squeue, or sbatch:

1. Verify ssh -G "$LUMI_SSH" resolves to a jump-routed target
2. Check for proxyjump or proxycommand
3. If it resolves as a direct login host, stop and fix SSH config before proceeding
4. Remember that lumi.csc.fi resolves to multiple login nodes (uan01–uan04) via DNS round-robin. A later SSH to the same alias may land on a different node, so tmux ls/ps are unreliable for monitoring long login-node processes; use durable shared-scratch artifacts as the source of truth.

Code Synchronization

• GitHub is the source of truth
• Commit and push outer/submodule changes, then pull on LUMI through the jump route
• Avoid scp/rsync for routine code sync
• Reserve file copy only for emergency patches or non-git artifacts

W&B Naming Norm

All LUMI launch scripts must follow the shared-project convention:

• One shared W&B project for comparison
• Slurm job name identical to WANDB_EXPERIMENT_NAME
• Group = <experiment-name>
• Run name = experiment name + backend/run id or timestamp
• Do not use alternate labels like rerun-* or dev-* unless W&B uses them too

GPU Job Template (Single Node)

#!/bin/bash
#SBATCH --account=project_<id>
#SBATCH --partition=small-g
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=56
#SBATCH --mem=480G
#SBATCH --time=3-00:00:00
#SBATCH --no-requeue
#SBATCH --open-mode=append
#SBATCH --output=%j.log
#SBATCH --error=%j.log

module load LUMI PyTorch/2.7.0-rocm-6.2.4-python-3.12-singularity-20250527

export NCCL_SOCKET_IFNAME=hsn0,hsn1,hsn2,hsn3
export NCCL_NET_GDR_LEVEL=3
export MIOPEN_USER_DB_PATH=/tmp/${USER}-miopen-cache
export MIOPEN_CUSTOM_CACHE_DIR=$MIOPEN_USER_DB_PATH
export ROCR_VISIBLE_DEVICES=$SLURM_LOCALID

srun singularity exec $SIFPYTORCH conda-python-simple -u train.py

Multi-Node Distributed Training

See references/pytorch-gpu-jobs.md for full multi-node templates with CPU binding masks and torch.distributed.run.

Partitions

Partition	Type	Max Walltime	Max Nodes	Billing
standard-g	Full-node GPU	2 days	1,024	nodes * 4 * hours GPU-h
small-g	Shared GPU	3 days	4	max(ceil(cores/8), ceil(mem/64GB), GCDs) * hours * 0.5
dev-g	Debug GPU	30 min--2 hrs	8--32	Same as small-g
standard	Full-node CPU	2 days	512	nodes * 128 * hours core-h
small	Shared CPU	3 days	4	max(cores, ceil(mem/2GB)) * hours
debug	Debug CPU	30 min	4	Same as small

Storage

Area	Path	Default Quota	Purpose
Home	/users/<username>	20 GB / 100k files	Config, scripts
Project	/project/<project>	50 GB (expandable to 500 GB)	Shared code, small data
Scratch	/scratch/<project>	50 TB (expandable to 500 TB)	Training data, checkpoints
Flash	/flash/<project>	2 TB (expandable to 100 TB)	Hot data (3x billing)
Object	LUMI-O (S3)	150 TB	Cold storage (0.25x billing)

Warning: No backups on any storage. /tmp on compute nodes is memory-backed (no local disk). Never install Python packages directly on Lustre — use containers + SquashFS.

Anti-Patterns

Anti-Pattern	Fix
Using CUDA_VISIBLE_DEVICES	Use ROCR_VISIBLE_DEVICES=$SLURM_LOCALID
--cpus-per-task=8 per GCD	Use 7 (only 56 cores available, not 64)
pip install on Lustre	Use container shell + make-squashfs
Missing MIOpen redirect	Set MIOPEN_USER_DB_PATH=/tmp/...
Missing NCCL vars	Set NCCL_SOCKET_IFNAME and NCCL_NET_GDR_LEVEL
Expecting conda activate to work	Use module load PyTorch/... + Singularity
No --no-requeue	Jobs auto-requeue on preemption, duplicating output
Using pre-Jan-2026 software	Recompile with PE 25.03+, ROCm 6.3.4
Wrong CPU targets in job	Add module load CrayEnv in sbatch script
Using zsh	Cray PE init scripts are bash-only. Use bash on LUMI.
Submitting small-g without dev-g probe	Always probe first, then chain with --dependency=afterok:
Using GPUs for raw dataset download/unpacking	Use CPU partitions (debug/small) for data prep
Direct SSH login without jump-node routing	Verify ssh -G shows proxyjump or proxycommand
scp/rsync for routine code sync	Use GitHub: commit, push, then pull on LUMI
Monitoring login-node tasks with tmux ls from a fresh SSH	lumi.csc.fi load-balances across uan01–uan04; use shared-scratch log/CSV/state files as the source of truth
Slurm job name differing from W&B experiment name	Use the same source variable for both
More than two concurrent dev-g probes	Chain probes with Slurm dependencies
Including --mail-type in sbatch	LUMI does not support Slurm email notifications
tmux: missing or unsuitable terminal	export TERM=xterm-256color before tmux. LUMI lacks terminfo for modern terminals (Ghostty, Kitty, etc.). Fix permanently via SetEnv TERM=xterm-256color in local ~/.ssh/config.

Detailed References

• references/hardware.md — Full hardware specs, GCD architecture, CPU-GPU affinity, network topology
• references/pytorch-gpu-jobs.md — Container workflow, multi-node templates, environment variables, venv extension

When This Skill Doesn't Have the Answer

If a LUMI-specific question isn't covered above or in the reference files, scrape the official documentation:

WebFetch url="https://docs.lumi-supercomputer.eu/" prompt="<your question>"

Key documentation sections:

• Hardware: https://docs.lumi-supercomputer.eu/hardware/
• Running jobs: https://docs.lumi-supercomputer.eu/runjobs/
• Software: https://docs.lumi-supercomputer.eu/software/
• PyTorch: https://docs.lumi-supercomputer.eu/software/packages/pytorch/
• Storage: https://docs.lumi-supercomputer.eu/storage/
• Post-maintenance updates: https://lumi-supercomputer.github.io/update-202601/
• User support: https://lumi-supercomputer.eu/user-support

After finding useful new information, consider updating this skill's reference files so future sessions benefit.

See Also

• slurm-gpu-training — General Slurm patterns (job submission, monitoring, preflight)
• gpu-training-acceleration — ROCm-compatible acceleration techniques