ab-test-design

# A/B Test Design ## When to Use **Use this skill when:** - A user wants to design a controlled experiment before shipping a product change -- including UI changes, copy changes, pricing changes, onboarding flow changes, algorithm changes, or notification strategies - A user needs to validate a product hypothesis with statistical rigor before committing engineering resources to a full rollout - A user asks about sample size calculation, test duration estimation, or minimum detectable effect selection for a split test - A user wants to compare two versions of an experience and needs a structured framework for control/variant definition, metric selection, and analysis pre-registration - A user needs to decide between two competing product directions using behavioral data rather than opinion or intuition - A user is setting up experimentation infrastructure and needs to understand the methodology behind traffic allocation, assignment mechanisms, and guardrail monitoring - A user wants to run a holdout test, a switchback test, or a geo-based experiment and needs guidance on the appropriate variant of experimentation methodology **Do NOT use this skill when:** - The user wants to analyze results from a test that has already run -- use a statistical analysis skill instead (t-tests, chi-squared, Mann-Whitney U, Bayesian posterior analysis) - The user needs to design a survey to collect attitudinal or qualitative data -- use `employee-survey` or `user-research-plan` - The user wants to rank or prioritize a backlog of features without testing -- use `prioritization-framework` - The user explicitly needs a multivariate test (MVT) with 3+ factors varying simultaneously -- MVT requires fractional factorial design, interaction effect modeling, and Taguchi methods that differ substantially from two-variant A/B methodology - The user is asking about causal inference from observational data without randomization -- that requires difference-in-differences, regression discontinuity, or instrumental variable methods - The user wants to test a change that cannot be isolated to a single variable (e.g., a full redesign touching layout, copy, and navigation simultaneously) -- recommend a sequential rollout or phased testing strategy instead - The user needs to interpret p-values or confidence intervals from existing results -- that is statistical output interpretation, not experiment design --- ## Process ### Step 1: Clarify the Business Question and Change Hypothesis Before writing anything, extract the minimum required inputs from the user. Do not proceed to design without understanding these fundamentals. - **What is the specific change being made?** Get surgical precision: "Change the H1 from 'Powerful analytics for your business' to 'See what's working in your business'" is actionable. "Improve the homepage" is not. - **What is the current state (control)?** This must be the existing production experience, not an idealized baseline. If the user cannot describe the control precisely, the test cannot be designed. - **Why does the team believe this change will work?** The reasoning should connect to a specific user behavioral theory -- cognitive load reduction, loss aversion framing, social proof activation, friction removal. If the team cannot articulate why, the test is exploratory (still valid, but hypothesis must reflect that). - **What is the unit of diversion?** This determines the randomization level: user-level (by user ID), session-level, device-level, or geo-level. User-level is default for most product experiments. Session-level is only appropriate for anonymous traffic where personalization is not possible, but it introduces carry-over contamination risk. - **What is the eligible population?** Not all users should be in the experiment. Identify who is eligible: new users only, logged-in users only, users who have reached a specific page, users on a specific platform (iOS vs. Android vs. web). Sample size is calculated against the eligible daily traffic, not total daily users. - **Are there any hard constraints?** Regulatory restrictions, seasonal events (Black Friday, year-end), contractual obligations, or in-flight engineering changes that make a clean test impossible. --- ### Step 2: Write a Falsifiable, Directional Hypothesis The hypothesis is not a wish. It is a structured prediction that can be proven false. - **Use the If-Then-Because structure without exception:** - **If** [specific, singular change to the experience] - **Then** [named metric] will [increase/decrease] by [X% relative or X percentage points absolute] - **Because** [behavioral mechanism that connects the change to the outcome] - **The quantified prediction (the "by X%") serves as the Minimum Detectable Effect (MDE).** If the team says "we expect a 20% lift," that sets the MDE. If the team says "we just want to see any improvement," default to a 5% relative MDE for high-volume metrics and 10-15% for low-volume metrics, and explain the trade-off. - **Do not use vague outcome language.** "Users will prefer the new design" is not a hypothesis. "7-day retention rate among new users will increase by 8% relative (from 25.0% to 27.0%)" is. - **Classify hypothesis type:** - **Optimization hypothesis:** "This specific change improves a known metric by X." High confidence, well-reasoned, narrow change. - **Exploratory hypothesis:** "We predict a directional positive effect but are uncertain about magnitude." Set a smaller MDE and plan to learn rather than strictly ship/no-ship. - **Defensive hypothesis:** "This change will not harm our primary metric." When making a change for technical or regulatory reasons, the test is a guardrail check rather than a lift test. - **Pre-commit to directionality.** If the hypothesis is directional (variant will be better), use a one-tailed test, which has more power but cannot detect unexpected negative effects. If there is genuine uncertainty about direction, use a two-tailed test. For product experiments, two-tailed tests are almost always the right default -- they detect harmful regressions. --- ### Step 3: Define Control and Variant with Surgical Precision Ambiguous variant definitions are the single most common source of test failure in practice. - **Document the control as the exact production state on the day the test design is written.** If the production experience changes during the test, the test is contaminated. Lock the control specification. - **Change exactly one variable between control and variant.** This is not a stylistic preference -- it is the logical precondition for causal attribution. If the button copy changes AND the button color changes, you cannot determine which drove the effect. - **For UI changes:** Provide or request a screenshot, Figma link, or component-level specification. "The button says 'Get Started'" is insufficient. Document: text, font weight, color hex code, size in px, placement relative to page elements, and any surrounding context that differs. - **For algorithm or ranking changes:** Specify the exact parameter change (e.g., "personalization model weights increased from 0.3 to 0.5 for recency signal"), the fallback behavior if the model fails, and whether users in the control group receive the old model output or a static baseline. - **For pricing or packaging changes:** Document the exact pricing displayed, any asterisks or fine print, the checkout flow downstream of the pricing page, and whether the discount or price difference is persistent through the funnel or only on the first screen. - **For email or notification experiments:** Document the exact subject line, preview text, send time, sender name, and whether the experiment is on open rate (subject line variable only) or click rate (body content variable) or downstream conversion (full funnel). - **Explicitly list what does NOT change.** This prevents implementation drift. "The following elements are identical in both variants: button color (#2563EB), button size (48px height), surrounding copy, page layout, header, footer, and all other interactive elements." --- ### Step 4: Select Primary Metric, Secondary Metrics, and Guardrail Metrics Metric selection is a design choice with major consequences. Getting this wrong invalidates the entire experiment. - **Primary metric rules:** - There is exactly one primary metric. One. Pre-registering multiple primary metrics requires a Bonferroni correction that dramatically reduces power. - The primary metric must be directly observable within the test duration -- a metric that takes 90 days to manifest cannot be primary for a 3-week test. - The primary metric must be causally connected to the change being made. If the change is on the pricing page, the primary metric should be trial signups, not 30-day revenue (which is too distant and high-variance). - Prefer rate metrics over absolute count metrics for primary. "Signup conversion rate" is better than "total signups" because it normalizes for traffic fluctuations. - **Primary metric hierarchy by test type:** - Landing page / CTA changes → conversion rate (visits-to-action) - Onboarding flow changes → activation rate (users completing key setup step), day-1 or day-3 retention - In-product feature changes → feature adoption rate, task completion rate, engagement depth (sessions per user, actions per session) - Search or recommendation algorithm changes → click-through rate (CTR), NDCG (normalized discounted cumulative gain) if ranking quality matters, dwell time - Pricing or packaging changes → trial-to-paid conversion rate, average revenue per user (ARPU), or plan selection distribution - Email or push notification changes → click rate (not open rate -- open rate is unreliable post-iOS Mail Privacy Protection), downstream conversion - **Secondary metrics:** 2-4 supporting metrics that provide mechanistic understanding. If signup rate goes up, does activation rate also go up, or did we attract lower-quality signups? Secondary metrics answer that question. - **Guardrail metrics:** These are hard stops. Define a threshold for each: - Page load time: must not increase by more than 200ms (P95 latency) - Error rate: must not increase by more than 0.5 percentage points - Core downstream funnel: the next conversion step must not decrease by more than 2% relative - Customer support tickets or complaint rate: must not increase - Revenue per user: even if not the primary metric, must not decrease significantly - Guardrail violations trigger an immediate test pause regardless of primary metric performance. - **Avoid vanity metrics as primary:** Page views, raw visit counts, and "engagement" without definition are not primary metrics. They can be secondary. - **For tests where the primary metric has high variance (revenue, order value):** Use a variance-reduction technique such as CUPED (Controlled-experiment Using Pre-Experiment Data), which uses pre-experiment baseline data for each user as a covariate to reduce variance by 30-70%, dramatically reducing required sample size. --- ### Step 5: Calculate Sample Size and Test Duration This is the most technically demanding step. Walk through it explicitly -- do not skip or estimate loosely. **Inputs required:** - **Baseline metric value (p):** The current control conversion rate or metric value from the last 14-30 days of production data. Do not use the all-time average -- use recent data because baselines drift. - **Minimum Detectable Effect (MDE):** The smallest effect size worth detecting. Expressed as relative change (10% relative lift on a 4% baseline = 0.4 percentage points absolute) or absolute change. Smaller MDE = larger required sample. The MDE should be grounded in business impact: "A 5% relative lift would generate $240K in additional ARR and is therefore worth detecting." - **Significance level (alpha):** 0.05 is standard (5% false positive rate). For tests with major business consequences or irreversible decisions, use 0.01. - **Statistical power (1 - beta):** 0.80 is the standard minimum. For critical tests or when false negatives are costly (e.g., a safety-critical feature), use 0.90. - **Number of variants:** For a standard A/B test (1 control, 1 variant), the sample size is calculated per variant. **Sample size formula for proportions (exact):** For a two-tailed test at alpha=0.05, power=0.80: ``` n = (z_alpha/2 + z_beta)^2 * (p1*(1-p1) + p2*(1-p2)) / (p1 - p2)^2 ``` Where: - z_alpha/2 = 1.96 (for alpha=0.05, two-tailed) - z_beta = 0.84 (for power=0.80) - p1 = baseline rate (control) - p2 = expected rate under variant (p1 + MDE_absolute) **Practical approximation for proportions:** ``` n ≈ 16 * p * (1-p) / d^2 ``` Where d = absolute MDE in proportion terms. This approximation is valid when p is between 0.05 and 0.50. **For continuous metrics (revenue per user, session length):** ``` n = 2 * (z_alpha/2 + z_beta)^2 * sigma^2 / d^2 ``` Where sigma is the standard deviation of the metric and d is the absolute MDE. This requires historical standard deviation data, not just a mean. **Duration calculation:** ``` Duration (days) = n_per_variant / (daily_eligible_traffic * allocation_fraction) ``` **Duration constraints:** - **Minimum: 7 days (1 full business week).** Day-of-week effects are real. A test that runs Monday-Wednesday will have biased results. - **Preferred minimum: 14 days (2 full weeks).** Captures two full business cycles and reduces novelty effect contamination. - **Maximum: 28-42 days.** Beyond 6 weeks, external factors (competitor launches, seasonality, media coverage) increasingly contaminate results. Users who joined partway through the test had different experiences. The experimental population shifts. - **If calculated duration exceeds 8 weeks:** The test is likely infeasible as designed. Options: increase MDE (accept only larger detectable effects), switch to a higher-funnel metric with more daily events, apply CUPED variance reduction, or abandon A/B testing in favor of a phased rollout with pre-post analysis (with appropriate caveats about confounding). - **Round up duration to the nearest full week** to ensure complete business cycles. **Novelty effect buffer:** For changes to UI elements users interact with daily (navigation, feed ranking, dashboard layout), add a novelty effect observation period. Plan to analyze the first 7 days separately from subsequent days. Novelty effects manifest as an initial spike in engagement that decays to a true steady-state effect. The steady-state result is the reliable signal. --- ### Step 6: Define Traffic Allocation and Assignment Mechanism - **Standard allocation: 50/50.** Equal split maximizes statistical power for a two-variant test. Only deviate from 50/50 if there is a specific reason (risk aversion, see below). - **Ramp-up allocation:** For high-risk changes (new checkout flow, pricing change, infrastructure-level change), start at 10/90 (10% to variant, 90% to control) for the first 48 hours while monitoring guardrail metrics. If guardrails hold, ramp to 50/50. Document this ramp-up in the analysis plan, as early data from the ramp period should not be included in the final analysis. - **Assignment unit:** User ID hash is the gold standard for logged-in products. Session ID for anonymous traffic. Cookie-based for anonymous web users (with awareness of cookie clearing contamination). Device ID for mobile apps. - **Stickiness:** Assignment must be deterministic and consistent. The same user ID must always resolve to the same variant. Implement via hash function: `variant = hash(user_id + experiment_id) % 100`. If the experiment ID is stable, assignment is stable. - **Mutual exclusivity:** If multiple experiments are running simultaneously, ensure users cannot be in two experiments that interact. An experiment on the CTA button and a simultaneous experiment on the pricing displayed on the same page will have interaction effects. Use a mutual exclusivity layer or a sequential experiment design. - **Exclusions to specify:** - Internal employees and QA users (they interact with the product artificially) - Bots and automated traffic (filter by user agent, session behavior analysis) - Users who are mid-funnel in a separate experiment that affects the same downstream metric - Users in geographic regions with regulatory restrictions on the change - New users vs. existing users: if the change is only relevant for one cohort, restrict eligibility --- ### Step 7: Write the Pre-Registered Analysis Plan The analysis plan must be written and locked before the test launches. Changes after seeing data are p-hacking by definition. - **Primary analysis method:** - For conversion rate (proportion metric): Two-proportion z-test or chi-squared test. Report the p-value, the confidence interval on the difference, and the relative lift. - For continuous metrics (revenue, session length): Two-sample t-test. Check for normality violations -- if the distribution is heavily right-skewed (as revenue data typically is), use a Mann-Whitney U test or apply a log transformation before the t-test. - For count metrics (actions per session, messages sent): Poisson regression or negative binomial regression if overdispersed. - **When to read results:** Set a single predetermined analysis date (when minimum sample size is reached and minimum duration is satisfied). Do not log into the dashboard daily. Early peeking -- even without acting -- inflates the Type I error rate significantly. If one analysis is pre-planned, alpha=0.05 is valid. If 5 peeks are taken, the true Type I error rate approaches 0.19. - **If sequential testing (multiple peeks) is required:** Use an alpha-spending function (O'Brien-Fleming boundary or Pocock boundary). This is acceptable but requires pre-registration of the number of interim analyses and the alpha-spending schedule. - **Decision rules (pre-register all four outcomes):** 1. **Ship variant:** Primary metric improves by >= MDE with p < alpha and no guardrail violations. 2. **Keep control:** Primary metric shows statistically significant degradation (p < alpha in wrong direction) OR any guardrail metric violates its threshold. 3. **Inconclusive -- extend:** Result is not significant but sample size was not fully reached (early termination scenario). Extend by a pre-specified increment (typically 1 additional week). 4. **Inconclusive -- conclude:** Minimum sample size and duration were reached. Result is not significant. The test has adequate power. Conclude "no detectable effect at the specified MDE." This is a valid result -- it means the change did not produce the effect hypothesized at the specified magnitude. Do not call it "neutral" and ship anyway. - **Segment analysis plan:** Pre-register any segments to be analyzed (mobile vs. desktop, new vs. returning users, geographic regions, pricing tiers). Post-hoc segment mining produces false positives at a high rate. If a segment analysis is not pre-registered, treat its result as a hypothesis for a future test, not a decision-making input. - **Novelty effect analysis:** Compare the primary metric for week 1 vs. week 2+ within the variant group. If the week 1 rate is substantially higher than week 2+, novelty effect is present and the week 2+ rate is the more reliable estimate. --- ### Step 8: Assemble the Test Documentation and Communication Plan - Write the full test design document using the Output Format below before any implementation begins. - Share the document with: engineering (for implementation verification), data/analytics (for logging verification), and product leadership (for awareness and stakeholder alignment). - **Implementation verification checklist:** - Logging is confirmed for the primary metric event, all secondary metrics, and all guardrail metrics - Assignment logging is confirmed (every assignment event writes to an experiment log with user_id, variant, timestamp) - The assignment is verified to be sticky (run a query: same user_id should never appear in both variant and control) - Baseline sample ratio mismatch (SRM) check is planned: the split should be within ~1% of the target ratio after 48 hours. If the control group has 52% of users and the variant has 48% when 50/50 was specified, there is a bug in the assignment mechanism. Do not analyze results until SRM is resolved. - **Stakeholder communication:** Set a calendar hold for the results readout date. Communicate the test duration and results date to stakeholders in advance. This prevents "when will we know?" pressure that causes premature peeking. --- ## Output Format ``` ## A/B Test Design: [Test Name] --- ### Test Overview | Field | Value | |--------------------|-------------------------------------------| | Test name | [Descriptive name: Component-Change-Goal] | | Experiment ID | [Internal tracking ID if applicable] | | Owner | [Name / Team] | | Status | [Draft / Ready for Review / Approved / Running / Complete] | | Target launch date | [Date] | | Results read date | [Date = launch + estimated duration] | | Estimated duration | [X weeks] | | Platform | [Web / iOS / Android / Email / Server-side] | | Eligible audience | [Description of eligible user population] | --- ### Business Context [1-3 sentences on why this test is being run now. What business problem or opportunity does it address? What will the team do with the result?] --- ### Hypothesis **If** [specific, singular change to the experience], **then** [named metric] will [increase/decrease] by [X% relative / X pp absolute] (from [current baseline] to [expected variant value]), **because** [behavioral mechanism connecting the change to the outcome]. **Hypothesis type:** [Optimization / Exploratory / Defensive] **Test directionality:** [Two-tailed (default) / One-tailed (justify reason)] --- ### Variants | Variant | Description | Implementation Notes | |----------------|----------------------------------------------|---------------------------| | Control (A) | [Current production experience -- precise description] | [Engineering notes] | | Variant (B) | [Modified experience -- precise description] | [Engineering notes] | **Single variable changed:** [Exactly one element that differs between A and B] **What is explicitly NOT changed:** [List of elements confirmed identical] --- ### Metrics