experiment-design

name	experiment-design
description	Use when the user wants to design experiments, plan ablation studies, structure baselines, or create incremental evaluation strategies. Triggers on phrases like "design ablation", "plan experiment", "what experiments should I run", "baseline comparison", or "experiment matrix".

Experiment Design Methodology

You are helping a researcher design rigorous experiments. Follow this methodology systematically.

Step 1: Understand the Research Question

Before designing any experiment:

• Ask what specific hypothesis or claim the experiment should support
• Identify the dependent variable (metric) and independent variables (factors)
• Clarify the baseline: what is the current best result or default configuration?

Step 2: Single-Variable Isolation

Every ablation study must change exactly ONE variable at a time. For each factor:

1. Define the factor — what is being varied (e.g., loss function, learning rate, architecture component)
2. List levels — all values this factor will take (e.g., CE, focal, VAR)
3. Fix everything else — document what stays constant (seed, data split, epochs, hardware)
4. Predict outcome — before running, state what you expect and why

Template for each ablation row:

| Run ID | Factor | Value | Fixed Config | Expected Outcome |
|--------|--------|-------|-------------|-----------------|

Step 3: Experiment Matrix

For multi-factor studies, use a structured matrix:

1. Full factorial — if factors are few (≤3) and levels are few (≤3 each)
2. Sequential elimination — if factors are many: run single-factor ablations first, then combine winners
3. Latin square — if full factorial is too expensive: sample representative combinations

Always calculate total runs before committing:

Total runs = product of all factor levels
GPU hours = total runs × hours_per_run

Step 4: Resource Estimation

For each experiment plan, estimate:

• GPU hours: runs × time_per_run (check with user's hardware)
• API costs: if using external APIs (Gemini, OpenAI), estimate tokens × price
• Wall clock time: accounting for sequential dependencies and GPU availability
• Storage: checkpoint sizes × number of runs

Flag if total cost exceeds reasonable bounds and suggest prioritization.

Step 5: Config Stub Generation

Generate configuration stubs that match the user's existing config format. Read existing configs first to match:

• File format (YAML, JSON, TOML)
• Key naming conventions
• Directory structure for outputs
• Logging/tracking integration (wandb, neptune, tensorboard)

Step 6: Execution Plan

Create a concrete execution plan:

1. Order runs by dependency (baselines first, then ablations)
2. Identify which runs can be parallelized across GPUs
3. Create a shell script or batch runner matching the project's existing patterns
4. Include checkpointing strategy for long runs

Step 7: Analysis Plan

Before running, define how results will be analyzed:

• Which metrics to compare (primary + secondary)
• Statistical significance test if applicable (paired t-test, bootstrap CI)
• How to handle failed/crashed runs
• Visualization: what plots to generate (comparison tables, bar charts, learning curves)

Verification Checkpoints

Before finalizing the experiment plan:

• Each ablation changes exactly one variable
• Baseline is clearly defined and will be run with same setup
• Resource estimate is within budget
• Config stubs match existing project format
• Analysis plan is defined before execution begins
• Seeds are fixed for reproducibility

Output Format

Always produce:

1. Experiment matrix table — all runs with their configurations
2. Resource estimate — GPU hours, API costs, storage
3. Execution script — ready-to-run commands matching project conventions
4. Analysis plan — metrics, comparisons, visualizations