メニュー
Check on the user's running experiments across all servers — progress, loss, ETA, stalled or crashed. Use for "how are my runs", "is training still going", or checking a specific experiment.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Find what's eating disk on full lab servers — old checkpoints, wandb and cache junk — and produce a deletion plan. Report-only; deletion needs per-path human approval. Use when a disk is full or the user asks to free space.
Compare experiment results across servers — differing hyperparameters and final metrics from local wandb data, with dashboard links. Use for "which ablation won" or comparing runs of a project.
Morning briefing across all lab servers — what finished or crashed overnight, which GPUs are free, which disks are filling up. Use when the user asks for a briefing, "what happened overnight", or overall lab/server status.
Launch a training run on the best lab server — picks a host by data locality and free GPUs, verifies code and environment, starts it in a detached tmux session. Use when the user wants to run, launch, or train something on the servers.
Reproduce a working project environment from one lab server on another — code at the same commit, a CUDA-compatible python env, and a smoke test. Use for "make this run on host B too" or setting up a new server.
Launch a hyperparameter sweep distributed across free GPUs on multiple lab servers, one detached tmux session per config. Use when the user wants to run several variants of an experiment at once.
| name | runs-status |
| description | Check on the user's running experiments across all servers — progress, loss, ETA, stalled or crashed. Use for "how are my runs", "is training still going", or checking a specific experiment. |
Find every live training run and report its health. Read-only.
list_hosts (per-host GPU utilization) and list_sessions on each
reachable host.
Candidate runs = tmux sessions with work-like names (train_*, sweep_*,
hopmux_*, project names) plus anything actually holding GPU memory:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader
Peek each candidate via run_command:
tmux capture-pane -p -t '<session>' -S -40
Extract the last progress line (epoch/step, loss, lr, ETA).
Stall check: output looks frozen AND the host's GPUs are at 0% util → capture again ~10s later; unchanged means STALLED.
Crash/finish classification: Traceback / CUDA OOM / NaN → CRASHED; completion or final-summary lines → FINISHED; otherwise RUNNING.
Table: host · session · state · last progress line · GPU ids/util. Then one line for each non-RUNNING entry: what happened and the first thing to try. Do not restart or kill anything from this skill.