hopmux

Find what's eating disk on full lab servers — old checkpoints, wandb and cache junk — and produce a deletion plan. Report-only; deletion needs per-path human approval. Use when a disk is full or the user asks to free space.

Compare experiment results across servers — differing hyperparameters and final metrics from local wandb data, with dashboard links. Use for "which ablation won" or comparing runs of a project.

Morning briefing across all lab servers — what finished or crashed overnight, which GPUs are free, which disks are filling up. Use when the user asks for a briefing, "what happened overnight", or overall lab/server status.

Launch a training run on the best lab server — picks a host by data locality and free GPUs, verifies code and environment, starts it in a detached tmux session. Use when the user wants to run, launch, or train something on the servers.

Reproduce a working project environment from one lab server on another — code at the same commit, a CUDA-compatible python env, and a smoke test. Use for "make this run on host B too" or setting up a new server.

Check on the user's running experiments across all servers — progress, loss, ETA, stalled or crashed. Use for "how are my runs", "is training still going", or checking a specific experiment.

Launch a hyperparameter sweep distributed across free GPUs on multiple lab servers, one detached tmux session per config. Use when the user wants to run several variants of an experiment at once.

Audit which code version each lab server has for a project — commits, dirty trees, divergence — and align them safely. Use when results differ between servers or before launching a sweep.

Find a dataset, checkpoint, or file across every lab server — locations, sizes, duplicates. Use for "where is coco", "which server has X", or locating checkpoints and project directories.

Find zombie processes holding GPU VRAM on lab servers (dead notebook kernels, orphaned trainers) and propose kill commands. Report-only. Use when GPUs look occupied but idle, or for "who is using GPU N".