design-experiment

name	design-experiment
description	Use when designing an A/B test or online controlled experiment to evaluate a product change
source	Kohavi, Tang & Xu "Trustworthy Online Controlled Experiments" (Cambridge University Press 2020); Evan Miller "How Not To Run an A/B Test" (evanmiller.org)
tags	["experimentation","ab-testing","statistics","product","data-science","hypothesis-testing"]
verified	true

Design Experiment

Design statistically sound A/B experiments that produce trustworthy causal evidence for product decisions.

Why This Is Best Practice

Adopted by: Microsoft (ExP platform, Kohavi's team), Google, Netflix, Airbnb — all maintain internal experimentation platforms Impact: Kohavi et al. report that only 1/3 of A/B experiments at Microsoft produce a positive result; without rigorous experiment design, teams ship features that feel successful but have no causal impact or actively harm metrics.

Why best: A/B testing is the only method that provides causal evidence in product development. Without it, correlation-based decisions (feature launched, DAU went up — success!) fail to account for confounders. Trustworthy experiments require pre-registration of hypotheses, power analysis, and fixed analysis windows.

Steps