| name | wsl-gpu-stepcovnet |
| description | Runs StepCOVNet GPU training and MERT extraction on Windows via WSL with TensorFlow CUDA libraries. Use for train_onset, overfit scripts, extract_mert_features with cuda, or when TensorFlow sees zero GPUs in WSL. |
| disable-model-invocation | true |
WSL GPU (StepCOVNet)
Windows development model
On Windows, CPU work stays in the clone's Windows venv; CUDA work runs in WSL. The repo clone can live on any drive or folder — dispatch code derives WSL paths from the current checkout, not from a fixed username or /mnt/c/Users/... string.
| Setting | Default | Override |
|---|
| WSL venv root | $HOME/stepcovnet-venv-wsl | WSL_VENV |
| WSL Python | $WSL_VENV/bin/python | STEPCOVNET_WSL_PYTHON |
| Repo root | cwd / script location | — |
Shell helpers share these via scripts/wsl_common.sh.
Rule
On native Windows, never run CUDA/GPU training in the Windows venv. From repo root (project venv activated):
python scripts/<script>.py <args>
when the script calls wsl_gpu.maybe_dispatch_for_training (or MERT --device=cuda dispatch).
Opt out: STEPCOVNET_NO_WSL=1.
GPU scheduling (one job at a time)
The WSL GPU is single-tenant for the agent:
| Rule | Detail |
|---|
| One training job | Do not start a second WSL GPU training run while one is already active (train, overfit, MERT extract with --device=cuda). |
| No train + infer overlap | Do not run GPU training and GPU inference (debug_*, --ar_decode, eval_dense_*, model.predict) in separate processes at the same time. |
| Before launching | assert_wsl_gpu_free_for_training() checks nvidia-smi and logs/gpu_wsl.lock (stepcovnet.wsl_gpu_lock). Windows dispatch acquires the lock before wsl; a second parallel launch fails on O_EXCL. Override: STEPCOVNET_FORCE_GPU=1. |
| Manual check | python -c "from stepcovnet import wsl_gpu; wsl_gpu.assert_wsl_gpu_free_for_training()" |
| After training | Offline decode/eval on GPU is fine sequentially once training has exited. |
CPU scripts (pytest, lint, config edits) may run while a GPU job is active — they do not use the WSL CUDA device.
Manual WSL (debugging or already inside WSL)
cd /path/to/your/clone
bash scripts/wsl_ensure_env.sh
source scripts/wsl_gpu_env.sh
export STEPCOVNET_IN_WSL=1
python scripts/<script>.py <args>
Without source wsl_gpu_env.sh, TensorFlow may silently fall back to CPU.
Onset event training
| Script | Purpose |
|---|
scripts/train_onset_event.py | Single experiment from JSON config |
scripts/run_overfit_tide_suite.py | Tide overfit: conv1d / mel / mert (--epochs=50) |
scripts/run_overfit_tide_ablations.py | Threshold sweep, loss/arch variants (MERT) |
scripts/run_overfit_tide_bisection.py | Diagnose, grid oracle, half-cheat ablations (EXP-11) |
scripts/debug_onset_overfit.py | Checkpoint diagnostics |
final_data: pass --training_index_path=data/final_data/training_index.json (or a smoke subset manifest). Do not rely on data_dir=val_data_dir alone when train/val must follow the P8 song split.
Configs: configs/overfit_tide/{conv1d,mel,mert}.json (default 50 epochs); configs/event/audio_baseline.json for multi-song event runs (num_queries / n_max_onsets = 2048).
Artifacts: models_wsl/overfit_tide/, models_wsl/overfit_tide_ablations/, MERT cache data/v2/test/tide.mert.npy.
Other GPU scripts
| Script | Track |
|---|
scripts/train_onset.py | Dense onset — use --training_index_path for final_data |
scripts/train_arrow.py | Arrow model |
scripts/extract_mert_features.py --device=cuda | MERT feature cache |
Already inside WSL
cd to repo root, bash scripts/wsl_ensure_env.sh if needed, source scripts/wsl_gpu_env.sh, export STEPCOVNET_IN_WSL=1, use python — do not nest another wsl call.
Console and logs
When running scripts under scripts/, follow scoped rule scripts-execution.mdc: visible terminal, TensorBoard up before/with training, captures under logs/, scratch under _tmp/.
