vr-ablation-study

name	vr-ablation-study
description	Analyzes existing experiment results and designs/executes ablation experiments to verify the contribution of each component.
user-invocable	true
argument-hint	["experiment-directory"]

Ablation Study

Performs ablation experiments to systematically verify the contribution of each component in an existing experiment.

Input

$ARGUMENTS - Experiment directory path (e.g., "experiments/experiment_20260305_llm_reasoning")

Optionally, a list of components to ablate can be specified.

Process

Step 1: Analyze Existing Experiment

• Read config.yaml to identify the experiment's components
• Read run.py to understand each component's role
• Check existing results in results/ to determine baseline performance

Step 2: Identify Ablation Targets

Automatically identify components that can be ablated:

• Model components: specific modules, layers, mechanisms
• Data components: preprocessing steps, data augmentation, specific features
• Training components: regularization, learning rate schedule, optimization techniques
• Hyperparameters: changes to key parameters

If the user specifies components, those components are prioritized.

Step 3: Design Ablation Conditions

Design ablation conditions for each component:

• Full model: original with all components included (baseline)
• -Component A: remove or deactivate A
• -Component B: remove or deactivate B
• -Component A, B: compound ablation (if necessary)

Generate config.yaml variants for each condition.

Step 4: Write and Run Experiment Code

• Reuse the existing run.py to write the ablation experiment code
• Run each condition independently
• Save results to the ablation/ directory

# ablation_run.py structure
ablation_conditions = [
    {"name": "full_model", "config": {...}},
    {"name": "without_component_a", "config": {...}},
    {"name": "without_component_b", "config": {...}},
]

for condition in ablation_conditions:
    results = run_experiment(condition["config"])
    save_results(f"ablation/{condition['name']}.json", results)

Step 5: Compare and Analyze Results

• Organize results from all conditions into a comparison table
• Quantify the contribution of each component
• Check statistical significance (when possible)

Step 6: Generate Figures

Visualize the ablation experiment results:

• Bar chart: performance comparison across conditions
• Grouped bar chart: comparison across multiple metrics
• Apply styles from scripts/plot_style.py

Output

Save results under the ablation/ subdirectory of the experiment directory.

File Structure

experiments/<experiment-dir>/
└── ablation/
    ├── ablation_config.yaml     # Ablation condition definitions
    ├── ablation_run.py          # Ablation experiment execution script
    ├── results/                 # Results per condition
    │   ├── full_model.json
    │   ├── without_component_a.json
    │   └── without_component_b.json
    ├── ablation_summary.md      # Results summary
    └── figures/
        ├── ablation_comparison.pdf
        └── ablation_comparison.png

ablation_summary.md Format

# Ablation Study: [Experiment Name]
Date: [YYYY-MM-DD]

## Ablation Conditions

| Condition | Description | Changes |
|-----------|-------------|---------|
| Full Model | Original model | - |
| -Component A | A removed | [Specific changes] |
| -Component B | B removed | [Specific changes] |

## Results

| Condition | Metric 1 | Metric 2 | Δ from Full |
|-----------|----------|----------|-------------|
| Full Model | X.XX | X.XX | - |
| -Component A | X.XX | X.XX | -X.XX |
| -Component B | X.XX | X.XX | -X.XX |

## Analysis
- Contribution of Component A: [Analysis]
- Contribution of Component B: [Analysis]
- Key findings: [Summary]

## Figures
- `figures/ablation_comparison.pdf`: [Description]

Notes

• Reuse existing run.py code as much as possible
• Each ablation condition changes only one variable to ensure fair comparison
• Confirm ablation conditions with the user before running experiments
• After results are available, use /vr-make-figures to generate additional figures or /vr-write-paper to incorporate into the paper
• Use for additional experiments requested during peer review