| name | experimental-design-ds |
| description | A/B testing, randomization, sample size calculation, confounding control, and causal inference for data science. Covers the full experimental lifecycle from hypothesis formulation through power analysis, randomization strategies, blocking, factorial designs, sequential testing, and the potential outcomes framework for causal claims. Use when designing experiments, planning A/B tests, calculating sample sizes, or reasoning about causation from data. |
| type | skill |
| category | data-science |
| status | stable |
| origin | tibsfox |
| modified | false |
| first_seen | "2026-04-12T00:00:00.000Z" |
| first_path | examples/skills/data-science/experimental-design-ds/SKILL.md |
| superseded_by | null |
Experimental Design for Data Science
Experimental design is the discipline of collecting data so that the analysis can answer the intended question. Ronald Fisher, working at the Rothamsted agricultural station in the 1920s, formalized the three pillars -- randomization, replication, and blocking -- that remain the foundation of every modern experiment, from clinical trials to A/B tests on websites. This skill covers experimental design from the data scientist's perspective: planning experiments, ensuring valid causal inference, and avoiding the pitfalls that invalidate conclusions.
Agent affinity: fisher (experimental design, ANOVA), tukey (exploratory analysis of experimental results), benjamin (ethical review of experiments)
Concept IDs: data-hypothesis-testing, data-confidence-intervals, data-probability-basics, data-sampling-methods
Fisher's Three Pillars
1. Randomization
Random assignment of experimental units to treatment groups ensures that any observed difference is either due to the treatment or due to chance -- not due to confounders. Without randomization, the groups may differ systematically in ways that bias the result.
Mechanism: Each unit is assigned to treatment or control by a random process (coin flip, random number generator). This does not guarantee balance -- it guarantees that imbalances are random and quantifiable by probability theory.
Why it works: Randomization breaks the association between treatment assignment and all potential confounders, both observed and unobserved. No observational adjustment method can do this for unobserved confounders.
2. Replication
Multiple experimental units per group. Replication serves two purposes:
- Statistical power: More units means smaller standard errors and greater ability to detect real effects.
- Generalizability: Results from one unit might be idiosyncratic. Results replicated across many units are more convincing.
Replication is not repetition. Measuring the same unit 10 times is repetition (estimates measurement error). Measuring 10 different units is replication (estimates treatment effect variability).
3. Blocking
Grouping experimental units by a known source of variability and randomizing within blocks. If you know that males and females respond differently to a treatment, block by sex and randomize within each block. This reduces unexplained variance and increases power.
Blocking vs. stratification: Same concept, different fields. Blocking is the experimental design term (Fisher); stratification is the sampling term.
The Experimental Lifecycle
