19-ab-test-setup-global

name	19-ab-test-setup-global
description	Use when the user wants a VALID experiment instead of a guess — hypothesis, one variable, sample size and runtime math, statistical significance, primary versus secondary metrics, multi-arm designs, and a results template, across Optimizely, VWO, and native Meta and Google tests. Trigger on 'A/B test', 'split test', 'how long should I run the test', 'is this result significant', 'test two versions', 'which creative is actually better'. Also use when a winner was declared after two days on tiny numbers. Not for — scaling the proven winner, see `55-scaling-ads-global`; analyzing data already collected, see `13-data-analysis-global`; auditing the account, see `21-ads-audit-global`.
metadata	{"version":"1.0.1","category":"performance","language":"en"}
license	MIT
triggers	["A/B test","split test","multivariate test","experiment design","statistical significance","sample size calculator"]
output	A .md file containing hypothesis, sample size calculation, primary/secondary metrics, test setup, timeline, and a results template ready for analysis
related	["product-marketing-context-global","13-data-analysis-global","03-performance-eval-global","21-ads-audit-global"]

A/B Test Setup (Global)

Run experiments that produce decisions, not noise. Most "A/B tests" in marketing are underpowered, peeked-at, and badly hypothesized — meaning the team learns nothing and ships the louder variant.

---

For Newbies

A valid A/B test answers one question: "Did this change cause a real improvement, or am I seeing noise?"

To answer it credibly you need four things:

1. A specific hypothesis with a numeric prediction
2. One variable changed (everything else identical)
3. Enough sample to detect the effect you care about
4. Statistical significance before you call a winner (typically p < 0.05)

If any one of these is missing, you don't have an A/B test — you have a coin flip with extra steps.

Common newbie mistake: running a test for 3 days, seeing variant B 40% higher, declaring victory, and shipping. Three days is too short to absorb day-of-week effects, and small samples produce wild swings. Variant B may revert (or reverse) by day 14.

---

Step 0 — Read Context

Read .agents/product-marketing-context.md if it exists. Audience size, average traffic, and current conversion rate determine whether a test is even feasible.

---

Step 1 — Information Gathering

Ask up to 4 questions:

1. What are you testing? (Ad headline / Landing page section / Email subject / Pricing display / CTA button / Creative video)
2. Primary metric? (CTR / Conversion rate / CPM / CPA / Revenue / Open rate / Reply rate)
3. Daily traffic to the test surface? (Needed for sample size and duration)
4. Goal of the test? (Lift X% on primary metric / Pick a winner among N candidates / Validate a strategic hypothesis)

---

The 7 Principles of a Valid A/B Test

1. Test exactly one variable

The cardinal rule. Change two things at once and you cannot attribute the result.

• Bad: "I changed the headline, the hero image, and the CTA color." → You learn nothing about which element drove the lift.
• Good: Change only the headline. Image, CTA, layout, traffic source, and audience targeting are identical.

If you must test multiple changes, use a multivariate test (MVT) — but those need much more traffic (often 4×–8× a single A/B).

2. Hypothesize with a number

"If we [change X], [metric Y] will increase by [Z%] because [reason]."