القائمة
Check on the user's running experiments across all servers — progress, loss, ETA, stalled or crashed. Use for "how are my runs", "is training still going", or checking a specific experiment.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Find what's eating disk on full lab servers — old checkpoints, wandb and cache junk — and produce a deletion plan. Report-only; deletion needs per-path human approval. Use when a disk is full or the user asks to free space.
Compare experiment results across servers — differing hyperparameters and final metrics from local wandb data, with dashboard links. Use for "which ablation won" or comparing runs of a project.
Morning briefing across all lab servers — what finished or crashed overnight, which GPUs are free, which disks are filling up. Use when the user asks for a briefing, "what happened overnight", or overall lab/server status.
Launch a training run on the best lab server — picks a host by data locality and free GPUs, verifies code and environment, starts it in a detached tmux session. Use when the user wants to run, launch, or train something on the servers.
Reproduce a working project environment from one lab server on another — code at the same commit, a CUDA-compatible python env, and a smoke test. Use for "make this run on host B too" or setting up a new server.
Launch a hyperparameter sweep distributed across free GPUs on multiple lab servers, one detached tmux session per config. Use when the user wants to run several variants of an experiment at once.
| name | runs-status |
| description | Check on the user's running experiments across all servers — progress, loss, ETA, stalled or crashed. Use for "how are my runs", "is training still going", or checking a specific experiment. |
Find every live training run and report its health. Read-only.
list_hosts (per-host GPU utilization) and list_sessions on each
reachable host.
Candidate runs = tmux sessions with work-like names (train_*, sweep_*,
hopmux_*, project names) plus anything actually holding GPU memory:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader
Peek each candidate via run_command:
tmux capture-pane -p -t '<session>' -S -40
Extract the last progress line (epoch/step, loss, lr, ETA).
Stall check: output looks frozen AND the host's GPUs are at 0% util → capture again ~10s later; unchanged means STALLED.
Crash/finish classification: Traceback / CUDA OOM / NaN → CRASHED; completion or final-summary lines → FINISHED; otherwise RUNNING.
Table: host · session · state · last progress line · GPU ids/util. Then one line for each non-RUNNING entry: what happened and the first thing to try. Do not restart or kill anything from this skill.