experiment-designer

name	experiment-designer
description	Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
compatibility	opencode

Experiment Designer

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

When To Use

Use this skill for:

• A/B and multivariate experiment planning
• Hypothesis writing and success criteria definition
• Sample size and minimum detectable effect planning
• Experiment prioritization with ICE scoring
• Reading statistical output for product decisions

Core Workflow

1. Write hypothesis in If/Then/Because format

• If we change [intervention]
• Then [metric] will change by [expected direction/magnitude]
• Because [behavioral mechanism]

2. Define metrics before running test

• Primary metric: single decision metric
• Guardrail metrics: quality/risk protection
• Secondary metrics: diagnostics only

3. Estimate sample size

• Baseline conversion or baseline mean
• Minimum detectable effect (MDE)
• Significance level (alpha) and power

Use:

python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute