vr-remote-experiment

name	vr-remote-experiment
description	Deploys experiments to remote machines (GPU servers, clusters) via SSH and automates execution/monitoring/result collection. Supports distributed execution across multiple machines by condition.
user-invocable	true
argument-hint	["experiment-directory"]

Remote Experiment

Automates deploying experiments to remote machines, executing them, monitoring, and collecting results.

Input

$ARGUMENTS - Experiment directory path (e.g., experiments/experiment_20260306_llm_reasoning)

The experiment directory must already contain run.py and config.yaml. (Write the code first with /vr-run-experiment, or prepare them manually.)

Process

Step 1: Load Remote Environment Configuration

Read the remote section of config.yaml. If the section does not exist, ask the user for machine information and add it to config.yaml.

Required information:

• SSH connection info (host, user, key_path, port)
• Remote working directory (work_dir)
• GPU ID allocation (gpu_ids)
• Python environment (conda_env or venv_path)
• Experiment condition distribution strategy

remote:
  machines:
    - name: gpu-server-1
      host: 192.168.1.100
      user: researcher
      key_path: ~/.ssh/id_rsa
      port: 22
      work_dir: /home/researcher/experiments
      gpu_ids: [0, 1]
      conda_env: research
    - name: gpu-server-2
      host: 192.168.1.101
      user: researcher
      key_path: ~/.ssh/id_rsa
      work_dir: /data/experiments
      gpu_ids: [0, 1, 2, 3]
      venv_path: /data/venvs/research

  distribution:
    strategy: round-robin   # round-robin | manual | by-condition
    conditions_per_machine: auto

Step 2: Deploy

Transfer experiment code to each remote machine.

# 1. Create remote directory
ssh user@host "mkdir -p <work_dir>/<experiment-name>"

# 2. Transfer experiment files (excluding results)
rsync -avz --exclude 'results/' --exclude '__pycache__/' \
  $ARGUMENTS/ user@host:<work_dir>/<experiment-name>/

# 3. Activate environment and install dependencies
ssh user@host "cd <work_dir>/<experiment-name> && \
  conda activate <env> && \
  uv sync"

# 4. Verify environment
ssh user@host "python --version && nvidia-smi --query-gpu=name,memory.free --format=csv"

Report the environment status of each machine to the user and confirm whether to proceed.

Step 3: Execute

Run experiments remotely via SSH. Use tmux to detach sessions so experiments continue even if the SSH connection drops.

# Single machine execution
ssh user@host "cd <work_dir>/<experiment-name> && \
  tmux new-session -d -s exp_<name> \
  'conda activate <env> && \
   CUDA_VISIBLE_DEVICES=0,1 python run.py 2>&1 | tee experiment_log.txt'"

Distributed Execution (Multiple Machines)

Distribute experiment conditions across machines:

• round-robin: assign conditions to machines sequentially
• manual: user specifies condition-machine mapping directly
• by-condition: specify via machine field in config.yaml conditions

# Pass conditions to run as environment variables per machine
ssh user@host1 "... EXPERIMENT_CONDITIONS='baseline,treatment_a' python run.py"
ssh user@host2 "... EXPERIMENT_CONDITIONS='treatment_b,treatment_c' python run.py"

Step 4: Monitor

Periodically check experiment progress.

# Check logs
ssh user@host "tail -20 <work_dir>/<experiment-name>/experiment_log.txt"

# Check GPU utilization
ssh user@host "nvidia-smi"

# Check tmux session status
ssh user@host "tmux has-session -t exp_<name> 2>/dev/null && echo 'RUNNING' || echo 'DONE'"

# Check process
ssh user@host "ps aux | grep run.py | grep -v grep"

Report the status of each machine to the user:

• Running / Completed / Error
• GPU memory usage
• Last log line (progress)
• Estimated remaining time (when possible)

Step 5: Collect Results

Once experiments on all machines are complete, collect results to local.

# Collect results from each machine into per-machine subdirectories
rsync -avz user@host1:<work_dir>/<experiment-name>/results/ \
  $ARGUMENTS/results/gpu-server-1/

rsync -avz user@host2:<work_dir>/<experiment-name>/results/ \
  $ARGUMENTS/results/gpu-server-2/

Merge results from multiple machines:

• Merge each machine's raw_results.json into a single raw_results.json
• Recalculate summary_stats.json based on the full dataset
• Preserve per-machine raw results in subdirectories

Step 6: Cleanup

After confirming with the user whether to clean up remote files:

# Clean up remote experiment directory (optional)
ssh user@host "rm -rf <work_dir>/<experiment-name>"

# Clean up tmux session
ssh user@host "tmux kill-session -t exp_<name> 2>/dev/null"

Output

File Structure

$ARGUMENTS/
├── remote_log.md               # Full remote execution log
├── results/
│   ├── raw_results.json        # Merged results
│   ├── summary_stats.json      # Merged summary statistics
│   ├── gpu-server-1/           # Per-machine raw results
│   │   ├── raw_results.json
│   │   └── experiment_log.txt
│   └── gpu-server-2/
│       ├── raw_results.json
│       └── experiment_log.txt
└── config.yaml                 # Including remote section

remote_log.md Format

# Remote Experiment Log: [Experiment Name]
Date: [YYYY-MM-DD]

## Machines
| Name | Host | GPUs | Status |
|------|------|------|--------|
| gpu-server-1 | 192.168.1.100 | 0,1 | completed |
| gpu-server-2 | 192.168.1.101 | 0-3 | completed |

## Distribution
- Strategy: round-robin
- gpu-server-1: baseline, treatment_a (conditions 1-2)
- gpu-server-2: treatment_b, treatment_c (conditions 3-4)

## Timeline
- [HH:MM] Deploy started
- [HH:MM] Deploy completed (all machines)
- [HH:MM] Experiments started
- [HH:MM] gpu-server-1 completed (duration: Xh Xm)
- [HH:MM] gpu-server-2 completed (duration: Xh Xm)
- [HH:MM] Results collected and merged

## Results Summary
[Merged results summary statistics]

## Errors
[Errors recorded, if any]

Notes

• Uses SSH key authentication (password authentication is not supported)
• Before deploying experiments, always confirm the machine list and condition distribution with the user
• After remote experiment execution, visualization can be done with /vr-make-figures $ARGUMENTS
• Use rsync's --partial --progress options to handle network instability
• SSH port may vary depending on the remote machine's firewall/security settings, so verify the port configuration
• In job scheduler environments such as SLURM or PBS, sbatch/qsub commands can be used instead
• Remote cleanup must always be confirmed with the user before proceeding