メニュー
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | ab-testing |
| description | You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results. |
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Check for product marketing context first:
If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
| Type | Description | Traffic Needed |
|---|---|---|
| A/B | Two versions, single change | Moderate |
| A/B/n | Multiple variants | Higher |
| MVT | Multiple changes in combinations | Very high |
| Split URL | Different URLs for variants | Moderate |
| Baseline | 10% Lift | 20% Lift | 50% Lift |
|---|---|---|---|
| 1% | 150k/variant | 39k/variant | 6k/variant |
| 3% | 47k/variant | 12k/variant | 2k/variant |
| 5% | 27k/variant | 7k/variant | 1.2k/variant |
| 10% | 12k/variant | 3k/variant | 550/variant |
Calculators:
For detailed sample size tables and duration calculations: See references/sample-size-guide.md
| Category | Examples |
|---|---|
| Headlines/Copy | Message angle, value prop, specificity, tone |
| Visual Design | Layout, color, images, hierarchy |
| CTA | Button copy, size, placement, number |
| Content | Information included, order, amount, social proof |
| Approach | Split | When to Use |
|---|---|---|
| Standard | 50/50 | Default for A/B |
| Conservative | 90/10, 80/20 | Limit risk of bad variant |
| Ramping | Start small, increase | Technical risk mitigation |
Considerations:
DO:
Avoid:
Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.
| Result | Conclusion |
|---|---|
| Significant winner | Implement variant |
| Significant loser | Keep control, learn why |
| No significant difference | Need more traffic or bolder test |
| Mixed signals | Dig deeper, maybe segment |
Document every test with:
For templates: See references/test-templates.md
Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.
1. Generate hypotheses (from data, research, competitors, customer feedback)
2. Prioritize with ICE scoring
3. Design and run the test
4. Analyze results with statistical rigor
5. Promote winners to a playbook
6. Generate new hypotheses from learnings
→ Repeat
Feed your experiment backlog from multiple sources:
| Source | What to Look For |
|---|---|
| Analytics | Drop-off points, low-converting pages, underperforming segments |
| Customer research | Pain points, confusion, unmet expectations |
| Competitor analysis | Features, messaging, or UX patterns they use that you don't |
| Support tickets | Recurring questions or complaints about conversion flows |
| Heatmaps/recordings | Where users hesitate, rage-click, or abandon |
| Past experiments | "Significant loser" tests often reveal new angles to try |
Score each hypothesis 1-10 on three dimensions:
| Dimension | Question |
|---|---|
| Impact | If this works, how much will it move the primary metric? |
| Confidence | How sure are we this will work? (Based on data, not gut.) |
| Ease | How fast and cheap can we ship and measure this? |
ICE Score = (Impact + Confidence + Ease) / 3
Run highest-scoring experiments first. Re-score monthly as context changes.
Track your experimentation rate as a leading indicator of growth:
| Metric | Target |
|---|---|
| Experiments launched per month | 4-8 for most teams |
| Win rate | 20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses) |
| Average test duration | 2-4 weeks |
| Backlog depth | 20+ hypotheses queued |
| Cumulative lift | Compound gains from all winners |
When a test wins, don't just implement it, document the pattern:
## [Experiment Name]
**Date**: [date]
**Hypothesis**: [the hypothesis]
**Sample size**: [n per variant]
**Result**: [winner/loser/inconclusive], [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight, e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]
Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.
Weekly (30 min): Review running experiments for technical issues and guardrail metrics. Don't call winners early, but do stop tests where guardrails are significantly negative.
Bi-weekly: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.
Monthly (1 hour): Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.
Quarterly: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?
Use when building or recreating a Planes-style presentation deck: a playback, pitch, demo, strategy or client deck in the studio's house design (the "big playback deck" look people liked). Also use when someone wants an on-brand HTML slide deck with Planes fonts, colours, transitions and layouts, or asks to "make a deck like the Lewis Silkin playback" or "build a deck in the Planes style".
Use when building or recreating a Planes-style presentation deck: a playback, pitch, demo, strategy or client deck in the studio's house design (the "big playback deck" look people liked). Also use when someone wants an on-brand HTML slide deck with Planes fonts, colours, transitions and layouts, or asks to "make a deck like the Lewis Silkin playback" or "build a deck in the Planes style".
Use when writing, rewriting, or reviewing any copy that should sound like Planes: pitch decks, proposals, playbacks, credentials and RFP responses, website and product copy, case studies, emails, taglines, headlines, or any client-facing or internal writing that needs to be on brand. Also use when copy reads generic, corporate, hypey, or off-voice and someone wants it to "sound like us" or "sound like Planes".
Use when writing, rewriting, or reviewing any copy that should sound like Planes: pitch decks, proposals, playbacks, credentials and RFP responses, website and product copy, case studies, emails, taglines, headlines, or any client-facing or internal writing that needs to be on brand. Also use when copy reads generic, corporate, hypey, or off-voice and someone wants it to "sound like us" or "sound like Planes".
Designs a collaborative workshop and builds it as a ready-to-run FigJam board, plus a facilitator run-sheet and a participant invite email. Use this skill whenever Fiona (or anyone at Planes) wants to plan, design, structure, or build a workshop, session, or facilitated meeting — and especially whenever FigJam is mentioned. Trigger on phrases like "create a workshop", "design a workshop", "build a FigJam board", "set up a workshop in FigJam", "I'm running a session with [client]", "plan a discovery workshop", "kickoff workshop", "ideation session", "retro board", "prioritisation workshop", "journey mapping session", "we need a workshop for X", "turn this into a FigJam", or any request to prepare a participatory session. Always use this skill for workshop design even if the person doesn't say "FigJam" explicitly — it produces the agenda, the board, the run-sheet, and the invite together so the whole session is ready to facilitate.
Designs a collaborative workshop and builds it as a ready-to-run FigJam board, plus a facilitator run-sheet and a participant invite email. Use this skill whenever Fiona (or anyone at Planes) wants to plan, design, structure, or build a workshop, session, or facilitated meeting — and especially whenever FigJam is mentioned. Trigger on phrases like "create a workshop", "design a workshop", "build a FigJam board", "set up a workshop in FigJam", "I'm running a session with [client]", "plan a discovery workshop", "kickoff workshop", "ideation session", "retro board", "prioritisation workshop", "journey mapping session", "we need a workshop for X", "turn this into a FigJam", or any request to prepare a participatory session. Always use this skill for workshop design even if the person doesn't say "FigJam" explicitly — it produces the agenda, the board, the run-sheet, and the invite together so the whole session is ready to facilitate.