hpc-submit

name	hpc-submit
description	Submit a long-running / large job to the configured SLURM HPC cluster over SSH, as a well-behaved shared-cluster citizen. Use when the user wants to run a heavy or long job on the cluster, queue a batch job, or send work to the cluster instead of running it locally or on a login/shared node. Cluster specifics (host, account, partitions, scratch dir) come from config/machine_profile.yaml.

Submit a job to the configured SLURM cluster

Submit batch work to the operator's HPC cluster over SSH using key-based auth, requesting only the resources the job needs so other cluster users aren't crowded out.

All cluster specifics live in config/machine_profile.yaml under hpc_offload.cluster: ssh_alias, login_host (+ fallback_login_hosts), slurm_account, scratch_root, partitions, and optional mail_user. Read that block FIRST; never hardcode another cluster's hostnames or accounts.

Connection

• Connect via the configured SSH alias (e.g. ssh <ssh_alias> 'hostname && whoami'). Key-based auth must already be set up; it should return instantly with no prompt. Never put a password in a command or file.
• If the connection fails: do NOT retry in a loop — repeated failed attempts can trip the cluster's intrusion blocking (fail2ban or similar) and ban the local IP for a while. Switch to a fallback_login_hosts entry instead, and wait before retrying the banned one.
• SLURM commands (sbatch, squeue, sinfo, sacct, scancel) run from the login node. Submit jobs through SLURM — never run heavy work directly on a login node or on shared interactive nodes.

Cluster environment

• Storage: put all working files, intermediate results, and large datasets under the configured scratch_root (typically fast, locally-attached storage such as /workdir/$USER or $SCRATCH). Don't run compute-heavy I/O against a network-mounted home directory.
• Software: discover available software with module avail and load what the job needs with module load lines in the job script's environment section. Record what the cluster exposes in manifests/hpc_software.json (the runner treats that manifest as authoritative — see agents/07).
• Containers: if the cluster provides a wrapped container runtime (e.g. a restricted Docker wrapper, Singularity/Apptainer), follow the cluster's own documentation; mounts are usually restricted to scratch.

Execution policy (important)

Do NOT auto-launch jobs, workflows, or containers. Default behavior: generate the batch script and show it to the user for review. Only run sbatch (or start a container/workflow engine) when the user explicitly says to submit it.

Procedure

1. Discover the partitions and current load before deciding on resources:

   ssh <ssh_alias> 'sinfo -o "%P %a %l %D %t %C"'   # partitions: avail, timelimit, nodes, state, cores
   ssh <ssh_alias> 'squeue --me'                     # the user's own jobs
   ssh <ssh_alias> 'squeue -o "%.18i %.12P %.8u %.2t %.10M %.6D %R"'  # cluster-wide load
   

   Choose the shortest partition (from hpc_offload.cluster.partitions) that comfortably fits the job's expected runtime — shorter jobs schedule sooner and are friendlier to the shared queue. Note how busy the partition is and tell the user what you found.

2. Decide resources conservatively. Start from hpc_offload.default_request in config/machine_profile.yaml and adjust for the job; do NOT request an entire node unless the user asks for it or the job demonstrably needs it.

3. Generate the batch script from templates/job.sbatch in this skill's directory, filling __JOBNAME__, __ACCOUNT__, __PARTITION__, __CPUS__, __MEM__, __TIME__, __WORKDIR__, __MAIL_USER__, and __COMMAND__. If mail_user is unset, delete the two --mail-* lines.

4. Stage inputs to scratch_root with rsync -av (never scp large trees file-by-file), show the script to the user, and submit only on their say-so.

5. After completion, retrieve outputs back to the local paths the caller expects (rsync from scratch), and report job state from sacct.