ab-testing-patterns

plugin: instantly updated: 2026-01-20 # A/B Testing Patterns ## Testing Fundamentals ### One Variable at a Time **CRITICAL:** Only change one element per test for clear attribution. | Test Type | Variable | Keep Same | |-----------|----------|-----------| | Subject Line | Subject only | Body, CTA, timing | | Opening Line | First sentence | Subject, rest of body | | CTA | Call to action | Subject, body intro | | Send Time | Delivery time | All copy elements | ### Sample Size Requirements | Confidence Level | Minimum Sample per Variant | |------------------|----------------------------| | 90% | 100 | | 95% (standard) | 150 | | 99% | 200 | **Formula:** ``` sample_size = (Z^2 * p * (1-p)) / E^2 Where: Z = 1.96 for 95% confidence p = expected conversion rate (use 0.5 if unknown) E = margin of error (typically 0.05) ``` ## Subject Line Testing ### Test Categories | Category | Control Example | Variant Example | |----------|-----------------|-----------------| | Curiosity vs Specific | "Quick question" | "2 min about {{company}}'s pipeline" | | Personal vs Generic | "{{first_name}}, saw this" | "Your team might like this" | | Question vs Statement | "Struggling with X?" | "How we fixed X for [Company]" | | Short vs Medium | "Quick win?" | "{{first_name}}, 2 ideas for {{company}}" | ### Best Practices 1. **Test 2-3 variants maximum** - More variants require more sample 2. **Run for minimum 3 days** - Account for daily patterns 3. **Test during stable periods** - Avoid holidays, major events 4. **Document everything** - Record hypothesis, results, learnings ## Body Copy Testing ### Elements to Test | Element | Low-Lift | High-Lift | |---------|----------|-----------| | Opening hook | Different pain point | Different approach entirely | | Social proof | Different company name | No social proof | | Value proposition | Reframe benefit | Different benefit | | CTA | Soft vs hard ask | Different action | ### Copy Frameworks to Test **PAS vs AIDA:** - PAS: Problem-Agitate-Solution (emotional) - AIDA: Attention-Interest-Desire-Action (logical) **Test Hypothesis:** PAS performs better for pain-point-heavy ICPs, AIDA for solution-seekers. ## Timing Tests ### Variables to Test | Variable | Options to Test | |----------|-----------------| | Day of week | Tue vs Thu (typically best) | | Time of day | 8-10am vs 2-4pm | | Timezone | Send in prospect's local time vs batch send | | Sequence gaps | 2-day vs 3-day follow-up gaps | ### Default Schedule (Starting Point) ``` Optimal Sending Windows: Primary: Tuesday-Thursday, 9-11am local time Secondary: Tuesday-Thursday, 2-4pm local time Avoid: Monday morning, Friday afternoon ``` ## Statistical Significance ### Quick Significance Check | Total Sample | Lift Needed for 95% Confidence | |--------------|--------------------------------| | 200 (100 per variant) | 15%+ lift | | 500 (250 per variant) | 10%+ lift | | 1000 (500 per variant) | 7%+ lift | ### Decision Framework ``` IF lift >= 15% AND sample >= 100/variant: Declare winner with medium confidence IF lift >= 10% AND sample >= 250/variant: Declare winner with high confidence IF lift < 10% OR sample < 100/variant: Continue test or call it inconclusive ``` ## Implementing A/B Tests in Instantly ### Method 1: Split Leads 1. Export lead list 2. Randomly split into Variant A and Variant B groups 3. Create two identical campaigns with one variable different 4. Use `move_leads_to_campaign` to assign leads ### Method 2: Sequential Testing 1. Run Control for X days, collect metrics 2. Update campaign with Variant (`update_campaign_sequence`) 3. Run Variant for X days, collect metrics 4. Compare (less rigorous, use only if lead volume is limited) ### Tracking Results ```markdown ## A/B Test Log **Test ID**: {uuid} **Campaign**: {campaign_name} **Variable**: {what_was_tested} **Hypothesis**: {expected_outcome} **Control**: - Version: {control_description} - Sample: {n} - Open Rate: {x}% - Reply Rate: {y}% **Variant**: - Version: {variant_description} - Sample: {n} - Open Rate: {x}% - Reply Rate: {y}% **Result**: {Winner|Inconclusive} **Lift**: {z}% **Confidence**: {confidence}% **Learning**: {what_we_learned} ```