| name | lab-brief |
| description | Morning briefing across all lab servers — what finished or crashed overnight, which GPUs are free, which disks are filling up. Use when the user asks for a briefing, "what happened overnight", or overall lab/server status. |
Lab brief
Produce a one-screen digest of every server hopmux knows. Optimize for "what
needs my attention", not completeness. Everything here is read-only.
Gather
-
list_hosts — per host note: reachable, gpusFree/gpusTotal, disks
(flag ≥90% full as urgent, ≥80% as worth a mention).
-
list_sessions on each reachable host — tmux sessions and recent agent
sessions.
-
For every tmux session that looks like work (training runs, sweep_*,
hopmux_* — not plain shells), peek at its tail with run_command:
tmux capture-pane -p -t '<session>' -S -60
Classify from the tail:
- CRASHED — Traceback, CUDA out of memory, NaN/nan loss, RuntimeError,
Killed, OOM
- FINISHED — completion messages, final-epoch summaries, wandb
"finished" lines, or the pane's process has exited
- RUNNING — progress bars / step-loss lines; capture twice ~5s apart if
unclear
- STALLED — output unchanged between captures AND that host's GPUs show
0% utilization
-
Optional: for crashed/finished runs whose project directory you know,
list_wandb_runs to attach dashboard URLs.
Report
Order by urgency; skip empty sections:
- Needs attention — crashes (host, session, one-line cause, last error
line), disks ≥90%, hosts that are usually up but unreachable.
- Finished overnight — host, session, final metric line if visible, wandb
link.
- Still running — host, session, last progress line (epoch/step/loss/ETA).
- Free compute — hosts with free GPUs (count + model), so the user knows
where the next job can go.
Rules: never restart, kill, or clean anything from this skill — report and
suggest. Keep the digest under ~30 lines; expand details only on request.
