vr-experiment-plan

name	vr-experiment-plan
description	Create a detailed experiment plan for a selected research idea. Define experiment variables, metrics, baselines, and datasets.
user-invocable	true
argument-hint	["idea-description or brainstorm-file-path"]

Experiment Planning

Develop a selected research idea into a detailed experiment plan.

Input

$ARGUMENTS - Research idea description or brainstorm file path

Process

Step 1: Idea Refinement

• Refer to the brainstorming results and elaborate on the selected idea
• Finalize the research question and hypothesis

Step 1.5: Baseline and Dataset Validation

Read .claude/skills/_shared/paper-search-protocol.md and execute a Level 1 search focused on: "standard baselines, commonly used datasets, and evaluation metrics for [research area]". Validate that the chosen baselines are current and no stronger recent baseline is being missed.

Step 2: Define Experiment Variables

• Independent Variables (IV): List of variables to manipulate
• Dependent Variables (DV): List of variables to measure
• Control Variables: List of variables to hold constant
• Experimental Conditions: Setting values for each condition

Step 3: Evaluation Metric Design

• Primary metrics
• Secondary metrics
• Measurement method and formula for each metric

Step 4: Baseline and Comparison Group Setup

• Select baseline methods
• Select comparison methods
• Unify conditions for fair comparison

Step 5: Data and Resource Planning

• List of required datasets
• Data collection/generation methods
• Required models and APIs
• Estimated number of API calls and costs

Step 6: Experiment Protocol

• Experiment execution order
• Estimated time for each step
• Error handling and retry policies
• Seed/configuration management for reproducibility

Output

Generate the following two files:

1. Experiment Plan: experiments/experiment_YYYYMMDD_<name>/plan.md

# Experiment Plan: [Experiment Name]
Date: [YYYY-MM-DD]

## Research Question
[Research question]

## Hypothesis
[Hypothesis]

## Variables
### Independent Variables
- [Variable name]: [Description] (Values: [v1, v2, ...])

### Dependent Variables
- [Metric name]: [Description] (Measurement method: [Description])

### Control Variables
- [Variable name]: [Fixed value]

## Experimental Conditions
| Condition | IV1 | IV2 | Description |
|-----------|-----|-----|-------------|
| baseline  | ... | ... | ...         |
| treatment | ... | ... | ...         |

## Metrics
### Primary
- [Metric]: [Formula/Description]

### Secondary
- [Metric]: [Formula/Description]

## Data Requirements
- [Dataset name]: [Source, size, purpose]

## Execution Plan
1. [Step]: [Description]
2. ...

## Expected Results
[Expected results for both cases: hypothesis confirmed and hypothesis rejected]

2. Experiment Configuration: experiments/experiment_YYYYMMDD_<name>/config.yaml

experiment:
  name: [Experiment name]
  date: [YYYY-MM-DD]
  description: [Description]

variables:
  independent:
    - name: [Variable name]
      values: [v1, v2, ...]
  dependent:
    - name: [Metric name]
      type: [float/int/categorical]

conditions:
  - name: baseline
    params: { ... }
  - name: treatment
    params: { ... }

execution:
  seed: 42
  n_trials: [Number of trials]
  timeout_per_trial: [Seconds]

output:
  results_dir: results/
  format: [json/csv]

Notes

• Consider the sample size needed to ensure statistical significance of the experiment
• Have the user review the experiment plan and incorporate feedback
• Once the plan is finalized, run the experiment with /vr-run-experiment <experiment-dir>