| name | launch-experiment |
| description | Launch a training run on the best lab server — picks a host by data locality and free GPUs, verifies code and environment, starts it in a detached tmux session. Use when the user wants to run, launch, or train something on the servers. |
Launch an experiment
Standard placement procedure. Ask only for what you cannot infer: the command
(or repo + script) and the dataset it needs.
Procedure
-
Data gravity first. list_datasets on candidate hosts — prefer a host
that already has the dataset. Copying terabytes is a last resort and needs
explicit user approval (then use copy_path / plan_migration).
-
Compute. list_hosts — among hosts with the data, find truly free GPUs
(inUse: false, i.e. <1.5GB VRAM held). Never place work on a GPU that is
holding someone's memory, even at 0% utilization. On shared hosts leave at
least one GPU free when you can.
-
Disk. The chosen host needs headroom for checkpoints — flag if the
target filesystem is over ~85%.
-
Code. Verify the project on that host with run_command:
git -C <dir> rev-parse --short HEAD && git -C <dir> status --porcelain | head
Dirty tree or missing repo → tell the user; don't silently train on unknown
code.
-
Env smoke test (adapt to the project's env activation):
cd <dir> && python -c 'import torch; print(torch.__version__, torch.cuda.is_available())'
-
Launch with start_session: the chosen host and dir, the command
prefixed with CUDA_VISIBLE_DEVICES=<ids>, and a short greppable session
name (e.g. train_<proj>_<mmdd>).
-
Verify it's alive ~20s later:
tmux capture-pane -p -t '<name>' -S -30
Confirm it got past imports and data loading. If it died, show the error
and fix or report.
Report
Host, GPU ids, tmux session name, code commit, and how to look at it (a hopmux
tab, or tmux attach -t <name>). If nothing was free: say which hosts free up
soonest (running sessions + their progress) — do NOT queue or wait silently.
