| name | experiment-designer |
| description | Manage — Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor. |
| executor | HYBRID |
| skill_id | business.product-team.experiment-designer |
| status | ADOPTED |
| security | {"level":"standard","pii":false,"approval_required":false} |
| anchors | ["business","testing","research","design"] |
| tier | 2 |
| input_schema | [{"name":"code_or_task","type":"string","description":"Code snippet, script, or task description to process","required":true}] |
| output_schema | [{"name":"plan","type":"string","description":"Strategic plan or design document"},{"name":"next_steps","type":"array","description":"List of recommended next steps"}] |
Experiment Designer
Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.
When To Use
Use this skill for:
- A/B and multivariate experiment planning
- Hypothesis writing and success criteria definition
- Sample size and minimum detectable effect planning
- Experiment prioritization with ICE scoring
- Reading statistical output for product decisions
Core Workflow
- Write hypothesis in If/Then/Because format
- If we change
[intervention]
- Then
[metric] will change by [expected direction/magnitude]
- Because
[behavioral mechanism]
- Define metrics before running test
- Primary metric: single decision metric
- Guardrail metrics: quality/risk protection
- Secondary metrics: diagnostics only
- Estimate sample size
- Baseline conversion or baseline mean
- Minimum detectable effect (MDE)
- Significance level (alpha) and power
Use:
python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute
- Prioritize experiments with ICE
- Impact: potential upside
- Confidence: evidence quality
- Ease: cost/speed/complexity
ICE Score = (Impact * Confidence * Ease) / 10
- Launch with stopping rules
- Decide fixed sample size or fixed duration in advance
- Avoid repeated peeking without proper method
- Monitor guardrails continuously
- Interpret results
- Statistical significance is not business significance
- Compare point estimate + confidence interval to decision threshold
- Investigate novelty effects and segment heterogeneity
Hypothesis Quality Checklist
