lab-brief

name	lab-brief
description	Morning briefing across all lab servers — what finished or crashed overnight, which GPUs are free, which disks are filling up. Use when the user asks for a briefing, "what happened overnight", or overall lab/server status.

Lab brief

Produce a one-screen digest of every server hopmux knows. Optimize for "what needs my attention", not completeness. Everything here is read-only.

Gather

1. list_hosts — per host note: reachable, gpusFree/gpusTotal, disks (flag ≥90% full as urgent, ≥80% as worth a mention).

2. list_sessions on each reachable host — tmux sessions and recent agent sessions.

3. For every tmux session that looks like work (training runs, sweep_*, hopmux_* — not plain shells), peek at its tail with run_command:

   tmux capture-pane -p -t '<session>' -S -60
   

   Classify from the tail:

   • CRASHED — Traceback, CUDA out of memory, NaN/nan loss, RuntimeError, Killed, OOM
   • FINISHED — completion messages, final-epoch summaries, wandb "finished" lines, or the pane's process has exited
   • RUNNING — progress bars / step-loss lines; capture twice ~5s apart if unclear
   • STALLED — output unchanged between captures AND that host's GPUs show 0% utilization

4. Optional: for crashed/finished runs whose project directory you know, list_wandb_runs to attach dashboard URLs.

Report

Order by urgency; skip empty sections:

• Needs attention — crashes (host, session, one-line cause, last error line), disks ≥90%, hosts that are usually up but unreachable.
• Finished overnight — host, session, final metric line if visible, wandb link.
• Still running — host, session, last progress line (epoch/step/loss/ETA).
• Free compute — hosts with free GPUs (count + model), so the user knows where the next job can go.

Rules: never restart, kill, or clean anything from this skill — report and suggest. Keep the digest under ~30 lines; expand details only on request.