ab-test-design-rituals

name	ab-test-design-rituals
description	Use when designing A/B tests. Covers hypothesis-driven testing, sample-size calculation, the test-design template, and the cultural rituals that make testing productive.

Ab Test Design Rituals

A/B tests answer questions. Run a test without a clear question and you'll have a clear non-answer.

Hypothesis template

We believe [change] will result in [outcome] for [audience], measured by [metric]. We'll know we're right if [primary metric] improves by [X%] within [Y days].

Sample size

Use a calculator (e.g., evanmiller.org). 5% baseline rate + 10% relative lift = ~7,000 visitors per arm.

Test duration

Minimum 1 week (captures day-of-week variation). Don't peek and stop early.

Test types

A/B (two variants). A/B/n (multiple variants — more samples required). Multivariate (combinations — even more).

Common mistakes

Stopping early when 'looks good'. Multiple metrics without primary. Underpowered tests (false negatives). 'Winning' variant that doesn't ship (no cultural follow-through).

Where this fits in X3 Compass

Applied across the X3 Compass design system.