| name | run-experiment |
| description | Plans and launches an hmfast experiment (training, inference, or scan) on the cluster with safe resource limits. Use when the user wants to run something that costs nontrivial compute, especially on GPU. |
Run experiment
Follow .claude/rules/cluster-usage.md and, for autonomous sessions, .claude/rules/autonomous-operation.md.
Pre-flight
- Identify the device. GPU partition? Which one? Quick
nvidia-smi check on the target node.
- Estimate memory. Sum array sizes in floats; multiply by
8 bytes (x64 default). Confirm it fits with headroom for JAX's default 75% preallocation. Show the arithmetic.
- Estimate runtime. Run a tiny version (small
N, few steps) and extrapolate. If the extrapolation is unclear, ask the user before launching the full run.
- Pick a seed and config. Save both into the output directory.
Launch
During and after
- Stream the log:
tail -f logs/<jobname>-<jobid>.out.
- On finish, verify saved arrays are finite and have the expected shapes.
- Write a
run.json manifest with: git hash, full config, seed, MaxRSS, wall time, output paths.
Stop conditions
If during the run you see OOM, repeated NaNs, or runtime exceeding the estimate by more than 2x, stop and report before resubmitting. Do not blindly retry with the same parameters.
