garnatxa-hpc

name

garnatxa-hpc

description

Use whenever the user is working on, connecting to, submitting jobs to, troubleshooting, or asking about the Garnatxa HPC cluster at I2SysBio (UV/CSIC, Valencia). Covers SSH/VPN setup, SLURM submission and monitoring (sbatch, srun, sacct, squeue, scancel, sinfo, plotjob, squeue_/sacct_), choosing the right QoS, writing efficient job scripts (serial, threaded, MPI, array, dependencies), the Lmod module system, mamba/conda environments, Singularity containers, Nextflow and Snakemake pipelines on SLURM, /home + /storage + /scr filesystems, tape archive via merlot/tapecopy, the self-hosted GitLab + VSCode workflow, usage policies and quotas, and the required acknowledgment in publications. Trigger this skill for any mention of Garnatxa, garnatxa.srv.cpd, i2sysbio HPC, or when a user is clearly working on a CSIC/UV HPC at I2SysBio — even when the user does not name the cluster explicitly but is asking about job scripts, SLURM, tape, GitLab on garnatxagitlab.uv.es, or VPN config files like i2sysbi.ovpn.

Garnatxa HPC Cluster — I2SysBio (UV/CSIC)

Garnatxa is a shared SLURM cluster at the Institute for Integrative Systems Biology (I2SysBio, joint UV/CSIC, Valencia). Use this skill to give the user concrete, copy-pasteable answers grounded in how Garnatxa is actually configured — not generic SLURM advice.

Cluster facts at a glance

• Login host: garnatxa.srv.cpd (SSH only). Prompt after login: [USERNAME@master ~]$.
• Scheduler: SLURM. Module manager: Lmod.
• External access: blocked. Connect from inside the UV network or via VPN (i2sysbio.ovpn with Garnatxa credentials, or vpn_uv_es.ovpn for UV staff).
• Tape access host: merlot (you ssh merlot from the login node).
• Self-hosted GitLab: https://garnatxadoc.uv.es/gitlab (LDAP tab, Garnatxa creds).
• Support / tickets: https://garnatxadoc.uv.es/support; email [e-mail].
• Compute: 14 compute nodes (cn00-cn13), 1232 hardware threads, mixed 64/80/128-CPU generations, ~18 TB RAM, ~20.5 TFLOPS. No GPUs — Garnatxa is a CPU-only cluster.
• Storage: CEPH (4.1 PB raw, ~3.5 PB usable); tape library is LTO-9 16 TB tapes (~720 TB online; /tape2/<TAPE_CODE>/ is only readable by the owning group).

Partitions and the QoS table — pick the right one or jobs sit forever

Partition	Use for	Time	Default mem	Notes
interactive	Interactive shells, light work	up to 1 d	4 GB	Use interactive command. Max 30 GB, 20 CPUs. Runs on nodes merlot, subirat.
global	All sbatch jobs	up to 15 d	2 GB	Pick QoS from table below. 14 compute nodes cn00-cn13 (1232 hardware threads, mixed 64/80/128-CPU).
tape	Tape archive ops (from merlot)	up to 7 d	–	tapecopy submits to this — you don't usually submit here yourself.

QoS	Max time	Max CPU (user)	Max RAM (user)	Priority
short	1 d	200	1300 GB	1000
medium	7 d	150	700 GB	750
long	15 d	100	360 GB	500
long-mem	15 d	80	1300 GB	250
extra	15 d	400	2800 GB	1000 (request via ticket only — same priority as short/interactive)

Per-user totals across QoS: 1000 running jobs, 5000 array tasks max. Per-QoS CPU/RAM caps are enforced (e.g. running on short caps you at 200 CPUs and 1300 GB).

Job-priority formula: AGE + FAIRSHARE + JOB SIZE + QOS PRIORITY. Heavy recent users get demoted by fairshare. extra is reserved for justified urgent jobs.

How to use this skill

This skill turns you into an operator on the cluster, not just an info source. If the user has SSH access configured locally (ssh garnatxa works without a password), reach into the cluster directly to diagnose and act, rather than asking the user to run commands and paste output back.

Canonical operator guide: read references/operator-loop.md first when you're asked to do anything on the cluster. It covers preflight, the read-only diagnostic loop, the submit/wait/diagnose pattern, pipeline mode, when to confirm, and recovery from common failures.

Default operating mode (summary)

1. Preflight every session using assets/preflight.sh — one SSH probe that verifies VPN, key auth, Garnatxa-only tooling, and the merlot hop. Map the failure mode to a fix instead of looping retries.
2. Reach into the cluster directly when it would help. Read-only diagnostics are safe to run without asking: squeue -u $USER, squeue_, sacct, sacct_, sinfo, sshare, sprio, scontrol show job, reading slurm-*.out/.err, module avail.
3. Confirm before destructive or shared-impact actions — scancel, rm -rf outside /tmp, large sbatch submissions, tape writes, shell config edits, GitLab pushes, anything on someone else's /storage/<group>.
4. Read only the reference file(s) you need. The references are self-contained deep dives; don't load them all eagerly.
5. Answer with concrete Garnatxa-specific commands and outputs — right hostnames, right partitions/QoS, right module load names — not generic SLURM/Linux. When writing a job script, start from an asset in assets/.
6. Push the user toward good citizenship: realistic resource requests (≥ 75 % CPU and memory efficiency), no heavy work on the login node, no Git pushes > 10 MB or with data files, no Docker (use Singularity), and the required acknowledgment in papers.

Operator-mode gotchas you must internalise

These come from running the skill against the real cluster — they bite agents who don't know about them:

• module is NOT available in non-interactive SSH. Wrap module calls in bash -lc: ssh garnatxa 'bash -lc "module load anaconda && mamba env list"'. Inside an sbatch script module works directly.
• /scr is Ceph-shared (not node-local) and is never auto-cleaned. Use /scr/$USER/ and clean up at end of job. $SCRATCH and $XDG_CACHE_HOME are both empty inside jobs — don't rely on them.
• /tape2 is only mounted on merlot. Tape work always starts with ssh merlot. tapecopy -l works for any user; ls /tape2/<CODE> only for the owning group.
• Compute-node SSH (ssh cn07 …) only works while you have a live job there. Nodes self-identify as osd00..osd13 (alias of cn00..cn13); resolve via scontrol show job <id> | grep NodeList=.
• scontrol show job <id> only finds running or very recently finished jobs. For historical jobs use sacct -j <id> -P --format=workdir%80.
• Log filenames are user-defined. Don't assume slurm-<id>.out — read the sbatch script's --output= or find near the WorkDir.
• sacct_ and squeue_ print harmless tput warnings under non-interactive SSH. Prepend TERM=xterm for clean output.
• source .../conda.sh; conda activate X 2>/dev/null || true is a trap. If X doesn't exist, activation silently fails and python3 quietly resolves to base (no error, just a ModuleNotFoundError later for packages you "know" are installed). Drop the || true, or verify the env exists first with conda env list.
• bash -lc poisons conda envs via the auto-loaded gcc module. The login shell auto-loads an old gcc module (e.g. gcc/9.4.0) that prepends /opt/ohpc/.../gcc/9.4.0/lib64 to LD_LIBRARY_PATH. Its ancient then shadows the newer one your conda env ships, and compiled extensions blow up with GLIBCXX_3.4.29' not foundssh garnatxa '/storage//envs/myenv/bin/python -c "import numpy"'ssh garnatxa 'bash -lc "...python..."'module purgeexport LD_LIBRARY_PATH=/lib:$LD_LIBRARY_PATHbash -lcmodule`; never wrap conda-env python in it.

Typical diagnostic loop (read-only, run freely)

ssh garnatxa '
  echo "=== queue ===";            squeue -u $USER --long
  echo "=== live efficiency ===";  squeue_ -u $USER
  echo "=== recent (7d) ===";      sacct -u $USER --starttime=$(date -d "-7 days" +%F) \
                                          --format=jobid,jobname%20,state,elapsed,reqcpu,reqmem,maxrss,exitcode
  echo "=== finished efficiency ==="; sacct_ -b -u $USER
  echo "=== fairshare ===";        sshare -u $USER
'

From that one round-trip you can diagnose: stuck PD jobs (look at REASON), under-utilization (CPU/MEM efficiency < 75 % → right-size), TIMEOUT loops (State=TIMEOUT repeatedly → move to higher QoS or split into checkpoints), fairshare depression (heavy recent use → expect longer queue waits).

Decision table

If the user is asking about…	Read
How to actually operate the cluster as an agent — preflight, submit/wait/diagnose loop, when to confirm, recovery patterns	references/operator-loop.md ← read this first
End-to-end playbooks for 14 common operations (run a pipeline, debug a failed array, archive a project, reset, etc.)	references/scenarios.md
Symptom → diagnostic → fix for ~20 failure modes (OOM, TIMEOUT, AssocGrpCpuLimit, htslib mismatch, …)	references/troubleshooting.md
First login, SSH config, password change, VPN, firewall, PuTTY, key setup	references/connecting.md
sbatch, srun, sacct, squeue, scancel, sinfo, plotjob, squeue_, sacct_, job templates, arrays, dependencies, MPI, OpenMP, interactive jobs, QoS choice, why a job is PD/queued	references/slurm.md
module commands, mamba/conda envs, Singularity/Podman, custom modulefiles, Docker (= "use Singularity"), the biotools bundle	references/software.md
Nextflow .config / .nf, Snakemake Snakefile / Snakeconfig.yaml, master/launcher sbatch for pipelines, work/ cleanup, DAG/report generation

Bundled assets

assets/ contains ready-to-edit job scripts, pipeline configs, and operator helpers the agent should use itself:

Operator helpers (the agent runs these):

• preflight.sh — one-shot SSH probe verifying VPN, key auth, Garnatxa-only tooling (squeue_, tapecopy, …), and the merlot hop.
• vpn_connect_macos.sh / vpn_disconnect_macos.sh — on macOS, bring the VPN up/down from the CLI when preflight shows it's down. Uses the Homebrew openvpn binary and collects the VPN credentials + admin password through native secure dialogs (nothing echoed), so the agent can drive the connection while the human just answers the popups. See references/connecting.md.
• wait_for_job.sh — poll a job to completion from the local side (the 8-hour SSH idle timeout breaks naive sbatch --wait).
• resubmit_with_bumped_resources.sh — read a finished job's MaxRSS / elapsed from sacct_, regenerate the sbatch with 1.5× headroom, ask for confirmation, submit.
• cleanup_pipeline.sh — cancel pipeline children first then the master, optionally purge work/ after confirm.
• ssh_config.template — ~/.ssh/config block with ControlMaster persistence + ProxyJump to merlot.

Job-script templates (the agent adapts these for the user):

• job_serial.sh — single-CPU job (short QoS).
• job_threaded.sh — OpenMP / multi-threaded job.
• job_mpi.sh — MPI job across nodes.
• job_array.sh — array job with one task per input.
• launcher_dependency.sh — orchestrator that runs a setup job then submits an array gated on its success.
• nextflow.config, nextflow_launcher.sbatch — Nextflow on the SLURM executor.
• snakemake_profile.yaml, snakemake_launcher.sbatch — Snakemake on the SLURM executor (via a per-user mamba env).

Hard rules the user is expected to follow

These come straight from the docs and are worth quoting back when relevant:

1. Don't treat Garnatxa like a workstation. Real work goes inside an sbatch job or an interactive session, not on the login node. Login-node processes that exceed ~30 minutes or ~8 GB RAM can be killed.
2. Don't request resources you won't use. Garnatxa emails actual CPU/mem usage after every job. Aim for ≥75% CPU efficiency and ≥75% memory efficiency. Use squeue_ / sacct_ / plotjob to verify.
3. Always set --time and --mem explicitly — defaults will kill long jobs. Use --mem-per-cpu when you want to scale memory with CPUs.
4. Pick a QoS that fits. short for ≤1 day, medium for ≤7 days, long for ≤15 days, long-mem for big-RAM. extra only by ticket.
5. No Docker. Use Singularity or Podman; Docker is intentionally not installed (daemon model is unsafe on a shared cluster).
6. No data in GitLab. The self-hosted GitLab is for source code only. Push size limit is 10 MB. Use .gitignore to keep data/, out/, ref/ out of the repo.
7. No backups. /home and /storage are not backed up. Use tape (tapecopy, after ssh merlot) for long-term archive; keep your own off-cluster copies for anything irreplaceable.
8. VPN before SSH unless you're inside the UV network.
9. Acknowledge Garnatxa in publications with the canonical text (see references/policies.md) and email a copy to [e-mail].

Inline cheat sheets

These are the most-asked items; full detail lives in references.

Submit, monitor, cancel

sbatch myjob.sh                   # submit
squeue -u $USER                   # what's queued / running
squeue_ -u $USER                  # + live CPU and memory efficiency
sacct  -u $USER --starttime=YYYY-MM-DD   # finished jobs since date
sacct_ -j JOBID                   # finished-job efficiency
plotjob -j JOBID -o cpu           # plot CPU efficiency (needs ssh -X)
plotjob -j JOBID -o mem -s        # save mem-efficiency plot to /tmp
scancel JOBID                     # cancel a job
scancel -u $USER                  # cancel ALL your jobs
sinfo                             # node/partition state

Minimum viable sbatch header

#!/bin/bash
#SBATCH --job-name=myjob
#SBATCH --output=myjob_%j.out
#SBATCH --error=myjob_%j.err
#SBATCH --qos=short              # short | medium | long | long-mem
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G                 # or --mem-per-cpu=
#SBATCH --time=06:00:00          # HH:MM:SS or D-HH:MM:SS
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=[e-mail]

module load <yourtool>
srun your_command

For OpenMP/threaded code, raise --cpus-per-task (keep --ntasks=1) and set export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK. For MPI, raise --ntasks and use mpirun -np $SLURM_NTASKS … after module load openmpi4.

Interactive session

interactive                       # 2 CPUs, 4 GB, 12 h
interactive -c 6 -m 30G -t 24:00:00   # 6 CPUs, 30 GB, 24 h

Mamba env

module load anaconda
mamba create -n myenv python=3
mamba activate myenv
mamba install -c bioconda samtools

Never mix module load <tool> and mamba activate <env> inside the same sbatch job — pick one source of binaries.

Move data on/off the cluster

# Local -> Garnatxa
scp -O ./file.txt [e-mail]:./
rsync --inplace --progress --partial --append -av ./mydir [e-mail]:.

# Garnatxa -> local
scp -O [e-mail]:./mydir/file.txt .

Archive to tape (LTO-9)

ssh merlot                       # tape ops happen on merlot, not master
tapecopy -l                      # list tapes and available space
tapecopy path/to/dir             # submits a SLURM job; check jobtape_*.out
tapecopy "out/*.sai"             # quote wildcards
# Recall:
cp /tape2/XXX006L9/home/user/test/out/file.sai /home/user/test/out

Communication style

• The audience is mostly researchers, not sysadmins. Use plain language, but keep commands exact — they will be copy-pasted.
• When asked "how do I run X?", produce a complete sbatch script, not a fragment, because users are often new to SLURM.
• When a user describes a problem (a stuck job, an OOM, a wrong output dir), diagnose with the Garnatxa-specific tooling first: squeue_ and sacct_ for efficiency, scontrol show job for stuck PDs, plotjob for over time.
• When unsure about a detail, point the user to the live docs at https://garnatxadoc.uv.es/ or to a support ticket — better than guessing.