vr-run-experiment

name	vr-run-experiment
description	Write and execute experiment code according to the experiment plan. Read config.yaml to automatically run the experiment and save results.
user-invocable	true
argument-hint	["experiment-directory"]

Run Experiment

Write and execute experiment code according to the experiment plan (plan.md, config.yaml).

Input

$ARGUMENTS - Experiment directory path (e.g., experiments/experiment_20260305_llm_reasoning)

Process

Step 1: Load Experiment Plan

• Read $ARGUMENTS/config.yaml
• Read $ARGUMENTS/plan.md
• Identify experimental conditions and variables

Step 2: Write Experiment Code

Generate the following files:

run.py - Main experiment script

"""
Auto-generated experiment script.
Runs the experiment according to config.yaml settings.
"""

Writing rules:

• Load config.yaml to read experiment parameters
• Run iteratively for each experimental condition
• Save results to the results/ directory
• Print progress (tqdm or print)
• Log errors and proceed to the next condition on failure
• Fix random seed to ensure reproducibility

utils.py - Utility functions (if needed)

Step 3: Install Dependencies

# Add any new dependencies to the project-level pyproject.toml
uv add <package>
uv sync

Step 4: Run Experiment

uv run python $ARGUMENTS/run.py

• Monitor errors that occur during experiment execution
• Modify code and re-run as needed
• Check intermediate results and alert the user if anomalies are detected

Step 5: Verify Results

• Confirm that result files were generated successfully
• Print basic statistics of the data (mean, standard deviation, min/max)
• Confirm no experimental conditions were missed

Output

Results File Structure

$ARGUMENTS/
├── results/
│   ├── raw_results.json       # Full raw results
│   ├── summary_stats.json     # Summary statistics
│   └── experiment_log.txt     # Execution log
└── run.py                     # Experiment script

results/raw_results.json Format

{
  "experiment": "Experiment name",
  "date": "YYYY-MM-DD",
  "config": { ... },
  "results": [
    {
      "condition": "Condition name",
      "trial": 1,
      "metrics": { "metric1": 0.85, "metric2": 0.72 },
      "metadata": { ... }
    }
  ]
}

results/summary_stats.json Format

{
  "by_condition": {
    "baseline": {
      "metric1": { "mean": 0.80, "std": 0.05, "min": 0.70, "max": 0.90, "n": 10 }
    },
    "treatment": {
      "metric1": { "mean": 0.85, "std": 0.04, "min": 0.78, "max": 0.92, "n": 10 }
    }
  }
}

Notes

• Inform the user of estimated time and cost before running the experiment
• Consider rate limiting for experiments that include API calls
• Implement checkpointing for long-running experiments
• After the experiment is complete, proceed with visualization using /vr-make-figures $ARGUMENTS
• If the experiment needs to be run on a remote GPU server, use /vr-remote-experiment $ARGUMENTS