| name | runs-status |
| description | Check on the user's running experiments across all servers — progress, loss, ETA, stalled or crashed. Use for "how are my runs", "is training still going", or checking a specific experiment. |
Runs status
Find every live training run and report its health. Read-only.
-
list_hosts (per-host GPU utilization) and list_sessions on each
reachable host.
-
Candidate runs = tmux sessions with work-like names (train_*, sweep_*,
hopmux_*, project names) plus anything actually holding GPU memory:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader
-
Peek each candidate via run_command:
tmux capture-pane -p -t '<session>' -S -40
Extract the last progress line (epoch/step, loss, lr, ETA).
-
Stall check: output looks frozen AND the host's GPUs are at 0% util →
capture again ~10s later; unchanged means STALLED.
-
Crash/finish classification: Traceback / CUDA OOM / NaN → CRASHED;
completion or final-summary lines → FINISHED; otherwise RUNNING.
Report
Table: host · session · state · last progress line · GPU ids/util. Then one
line for each non-RUNNING entry: what happened and the first thing to try.
Do not restart or kill anything from this skill.
