| name | ab-test-setup |
| description | Manage — When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions 'A/B test,' 'split test,' 'experiment,' 'test this change,' 'variant co |
| license | MIT |
| metadata | {"version":"1.0.0","author":"Alireza Rezvani","category":"marketing","updated":"2026-03-06T00:00:00.000Z"} |
| executor | HYBRID |
| skill_id | business.marketing-skill.ab-test-setup |
| status | ADOPTED |
| security | {"level":"standard","pii":false,"approval_required":false} |
| anchors | ["business","testing","research","design"] |
| tier | 2 |
| input_schema | [{"name":"code_or_task","type":"string","description":"Code snippet, script, or task description to process","required":true},{"name":"context","type":"string","description":"Additional context or background information","required":false}] |
| output_schema | [{"name":"plan","type":"string","description":"Strategic plan or design document"},{"name":"next_steps","type":"array","description":"List of recommended next steps"}] |
A/B Test Setup
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Initial Assessment
Check for product marketing context first:
If .claude/product-marketing-context.md exists, read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
- Test Context - What are you trying to improve? What change are you considering?
- Current State - Baseline conversion rate? Current traffic volume?
- Constraints - Technical complexity? Timeline? Tools available?
Core Principles
1. Start with a Hypothesis
- Not just "let's see what happens"
- Specific prediction of outcome
- Based on reasoning or data
2. Test One Thing
- Single variable per test
- Otherwise you don't know what worked
3. Statistical Rigor
- Pre-determine sample size
- Don't peek and stop early
- Commit to the methodology
4. Measure What Matters
- Primary metric tied to business value
- Secondary metrics for context
- Guardrail metrics to prevent harm
Hypothesis Framework
Structure
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Example
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
Test Types
| Type | Description | Traffic Needed |
|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combinations | Very high |
| Split URL |
