launch-experiment

name	launch-experiment
description	Launch a training run on the best lab server — picks a host by data locality and free GPUs, verifies code and environment, starts it in a detached tmux session. Use when the user wants to run, launch, or train something on the servers.

Launch an experiment

Standard placement procedure. Ask only for what you cannot infer: the command (or repo + script) and the dataset it needs.

Procedure

1. Data gravity first. list_datasets on candidate hosts — prefer a host that already has the dataset. Copying terabytes is a last resort and needs explicit user approval (then use copy_path / plan_migration).

2. Compute. list_hosts — among hosts with the data, find truly free GPUs (inUse: false, i.e. <1.5GB VRAM held). Never place work on a GPU that is holding someone's memory, even at 0% utilization. On shared hosts leave at least one GPU free when you can.

3. Disk. The chosen host needs headroom for checkpoints — flag if the target filesystem is over ~85%.

4. Code. Verify the project on that host with run_command:

   git -C <dir> rev-parse --short HEAD && git -C <dir> status --porcelain | head
   

   Dirty tree or missing repo → tell the user; don't silently train on unknown code.

5. Env smoke test (adapt to the project's env activation):

   cd <dir> && python -c 'import torch; print(torch.__version__, torch.cuda.is_available())'
   

6. Launch with start_session: the chosen host and dir, the command prefixed with CUDA_VISIBLE_DEVICES=<ids>, and a short greppable session name (e.g. train_<proj>_<mmdd>).

7. Verify it's alive ~20s later:

   tmux capture-pane -p -t '<name>' -S -30
   

   Confirm it got past imports and data loading. If it died, show the error and fix or report.

Report

Host, GPU ids, tmux session name, code commit, and how to look at it (a hopmux tab, or tmux attach -t <name>). If nothing was free: say which hosts free up soonest (running sessions + their progress) — do NOT queue or wait silently.