| name | slurm-hpc |
| description | Slurm/HPC workflow helper. Use when working on clusters, partitions, nodes, sbatch/srun jobs, GPU/CPU allocations, queues, logs, checkpoints, or any resource-heavy command — e.g. "submit the job", "check the queue", "잡 돌려줘", "큐 확인", "GPU 몇 장", "학습 돌려줘" on a cluster; also digesting a long job log (job-digest, "로그 요약") and reconciling finished Slurm jobs into shared state (run-reconcile, sacct). Reads local cluster reference when generated.
|
Default: protect login nodes. Ask before expensive or unclear resource use.
Rules
- If Slurm commands exist, prefer
sbatch for long jobs and allocated srun for interactive work.
- Never run long/GPU/high-CPU jobs directly on login nodes.
- Confirm partition, account, time, GPU/CPU/mem, output path before heavy jobs.
- Job scripts should use
set -euo pipefail, cd "$SLURM_SUBMIT_DIR", and %x-%j logs.
- Python jobs should use project-local commands, usually
uv run.
- Track job id, command, config, seed, commit, checkpoint, and log path.
Local Cluster Reference
If present, read references/cluster.generated.md before suggesting partitions,
nodes, GPU types, limits, or default Slurm resources.
If missing, ask the user to run:
~/.oh-my-setting/scripts/generate-slurm-skill.sh
Useful Commands
sinfo
squeue -u "$USER"
scontrol show partition
scontrol show node <node>
sacct -j <job_id>
scancel <job_id>
Reconciling Async Jobs
A long Slurm job outlives the agent session that launched it. The run ledger
records the launch (with slurm_job_id), but not the terminal state, so
"is it done? did it OOM?" goes stale and agents relaunch duplicates. Reconcile
recorded jobs against sacct/squeue and write the outcome back so the next
agent — any of the three — sees ground truth.
~/.oh-my-setting/scripts/run-reconcile.sh scan
~/.oh-my-setting/scripts/run-reconcile.sh apply --memory
~/.oh-my-setting/scripts/run-reconcile.sh list
It only records terminal jobs (COMPLETED/FAILED/TIMEOUT/OOM/…), is idempotent
(skips already-reconciled ids), and saves a job-digest summary per job under
.oms/runs/reconcile/. Run it on session start when work may have finished
while you were away.
