sweep

name	sweep
description	Launch a hyperparameter sweep distributed across free GPUs on multiple lab servers, one detached tmux session per config. Use when the user wants to run several variants of an experiment at once.

Distributed sweep

Turns one experiment plus a parameter grid into k placed runs. This reuses the launch-experiment checks, then multiplies them.

1. Enumerate configs from the user's grid. Show the full list; if k > 8, get an explicit go-ahead before launching.
2. Build the placement pool. list_hosts → truly free GPUs (inUse: false) on hosts that pass the launch checks:
   • dataset present (list_datasets)
   • code at the SAME commit on every target host (git -C <dir> rev-parse --short HEAD via run_command; refuse mixed commits unless the user accepts the skew)
   • env smoke test per host
   • disk headroom for k sets of checkpoints Etiquette: leave ≥1 GPU free on shared hosts; never take a GPU that is holding someone's VRAM.
3. Launch each config with start_session (background): session name sweep_<proj>_<tag>_<i>, command prefixed CUDA_VISIBLE_DEVICES=<id>, and a distinct output/checkpoint dir per config (inject a run tag) so runs can't clobber each other.
4. Verify all alive ~30s later — tmux capture-pane -p -t '<name>' -S -20 for each; relaunch or report failures.

Report

The placement map: config · host · GPU · session · output dir — plus "check on them later with /runs-status". If the pool is smaller than k: place what fits and list the remainder explicitly — do NOT silently queue.