| name | product-analytics |
| description | Product analytics and experimentation frameworks. Use for defining metrics, designing A/B tests, analyzing funnels and cohorts, measuring feature impact, and setting up analytics infrastructure. |
| metadata | {"tags":["product-analytics","metrics","ab-testing","experimentation","funnel","cohort","statistics"]} |
Product Analytics
Data-driven product decisions through metrics, experimentation, and behavioral analysis.
"In God we trust, all others bring data." — W. Edwards Deming
1. Metrics Framework
North Star Metric (NSM)
The single metric that best captures the core value your product delivers.
Characteristics:
- Measures customer value (not revenue)
- Reflects the core JTBD outcome
- Leading indicator of sustainable growth
- Actionable by product teams
Examples:
| Product | NSM | Why |
|---|
| Airbnb | Nights booked | Core value exchange |
| Spotify | Time spent listening | Content engagement |
| Slack | Messages sent | Communication value |
| Notion | Weekly active docs | Organized work |
| Duolingo | Lessons completed | Learning progress |
Anti-patterns:
- Revenue as NSM → optimizes for extraction, not value
- Downloads as NSM → optimizes for vanity, not retention
- Page views as NSM → optimizes for engagement, not outcomes
Input → NSM → Output
Input Metrics (leading, actionable)
├─ Acquisition: Sign-up rate, activation rate
├─ Engagement: Feature adoption, session frequency
└─ Quality: NPS, support ticket rate
↓
North Star Metric (health indicator)
↓
Output Metrics (lagging, business)
├─ Revenue: ARR, MRR, LTV
├─ Retention: Logo retention, revenue retention
└─ Efficiency: CAC, payback period
The AARRR Framework (Pirate Metrics)
Acquisition → Activation → Retention → Revenue → Referral
(A) (A) (R) (R) (R)
Acquisition: Users find you (SEO, ads, content, viral)
Activation: Users experience core value (first "aha" moment)
Retention: Users come back (Day-1, Day-7, Day-30)
Revenue: Users pay (conversion, expansion, upsell)
Referral: Users invite others (viral coefficient, NPS)
Key ratios:
- Activation rate: % of sign-ups who reach "aha" moment
- Retention rate: % of activated users who return
- Conversion rate: % of retained users who pay
- Viral coefficient (K): avg referrals per user; K > 1 = exponential growth
2. Funnel Analysis
Building a Funnel
Step 1: Define the critical path
"What are the minimum steps a user must take to get value?"
Step 2: Measure conversion at each step
Step Users Conversion Drop-off
─────────────────────────────────────────────
Landing 10,000 100% —
Sign-up 3,000 30% 70% ← Biggest drop
Onboarding 1,500 50% 50%
First Value 750 50% 50%
Day-7 Return 375 50% 50%
Paid 75 20% 80%
Analysis:
- Focus on the biggest drop-off first
- A 10% improvement at a 70% drop-off beats a 50% improvement at a 10% drop-off
- Segment funnel by channel, persona, or cohort
Funnel Segmentation
Funnel by Channel:
Channel Landing Sign-up Activation Retention
─────────────────────────────────────────────────────
Organic 5,000 1,500 750 375
Paid 4,000 800 240 72
Referral 1,000 700 490 343
Insight: Referral has highest quality (49% activation vs 16% paid)
Action: Shift budget from paid to referral program
3. Cohort Analysis
Cohort Retention Table
Cohort Sign-up Week 0 Week 1 Week 2 Week 3 Week 4 Week 8
─────────────────────────────────────────────────────────────────────
2026-01-01 100% 45% 38% 35% 33% 30%
2026-01-08 100% 48% 41% 37% 35% —
2026-01-15 100% 50% 43% 39% — —
2026-01-22 100% 52% 45% — — —
2026-01-29 100% 55% — — — —
Signals:
- Flattening curve (horizontal at Week 4+): Product-Market Fit signal
- Improving cohorts (each row better than last): Product improving
- Declining cohorts: Product degrading or market shifting
- Benchmarks:
- Consumer SaaS: 20-25% Week-4 retention = good, 30%+ = excellent
- B2B SaaS: 40-50% Week-4 retention = good, 60%+ = excellent
- Mobile apps: 15-20% Day-30 retention = good
Cohort Analysis by Feature
Cohort that used Feature X vs didn't:
Week 1 Week 2 Week 4
─────────────────────────────────────────
Used Feature X 60% 55% 50%
Didn't use 35% 25% 15%
Lift +71% +120% +233%
Insight: Feature X is strongly correlated with retention
Action: Prioritize onboarding users to Feature X
4. A/B Testing & Experimentation
Experiment Design
1. Hypothesis
"We believe [change] will [impact metric] by [amount] because [reason]"
2. Success Criteria
Primary metric: The one metric that determines go/no-go
Secondary metrics: Guardrails (don't hurt X while improving Y)
Guardrail metrics: Must not degrade (e.g., page load time)
3. Randomization
- Truly random assignment (check for bias post-hoc)
- Stable (same user always sees same variant)
- Consistent across devices
4. Duration
- Minimum: 1 full business cycle (7 days for weekly patterns)
- Typical: 2 weeks
- Maximum: Set before starting; don't run indefinitely
Sample Size Calculation
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize
baseline = 0.20
mde = 0.03
alpha = 0.05
power = 0.80
effect_size = proportion_effectsize(baseline + mde, baseline)
analysis = NormalIndPower()
sample_size = analysis.solve_power(
effect_size=effect_size,
alpha=alpha,
power=power,
ratio=1
)
Quick reference:
| Baseline | MDE | Sample/variant |
|---|
| 10% | +2% | ~7,000 |
| 20% | +3% | ~3,500 |
| 30% | +5% | ~1,800 |
| 50% | +5% | ~1,600 |
Statistical Analysis
from scipy import stats
control_visitors = 3500
control_conversions = 700
variant_visitors = 3500
variant_conversions = 875
z_stat, p_value = stats.proportions_ztest(
[control_conversions, variant_conversions],
[control_visitors, variant_visitors]
)
lift = (variant_conversions/variant_visitors - control_conversions/control_visitors)
Decision framework:
| p-value | Practical Significance | Decision |
|---|
| < 0.05 | Yes (lift > MDE) | ✅ Implement |
| < 0.05 | No (lift < MDE) | ⚠️ Monitor, not implement |
| > 0.05 | Yes | ⚠️ Run longer or accept risk |
| > 0.05 | No | ❌ No effect, don't implement |
Early Stopping & Peeking
The Problem: Peeking at results and stopping when p < 0.05 inflates false positive rate from 5% to ~30%.
Solutions:
- Fixed duration: Set end date before starting, don't look until then
- Sequential testing: Use proper sequential boundaries (more complex)
- Accept uncertainty: Some experiments need to run longer
Experiment Checklist
5. Feature Impact Measurement
Before/After Analysis
Feature Launch Impact:
Metric Before After Δ Significant?
────────────────────────────────────────────────────────
Activation 25% 28% +3pp Yes (p=0.02)
Day-7 Retention 18% 19% +1pp No (p=0.18)
Revenue/user $12 $14 +$2 Yes (p=0.01)
Interpretation: Feature improved activation and revenue but not retention.
Next step: Investigate why retention didn't improve.
Holdout Groups
Reserve a small % of users (5-10%) who never see the new feature:
Benefits:
- Measure long-term impact (some features have delayed effects)
- Detect negative interactions between features
- Provide clean baseline for future experiments
Trade-off: Slower to detect impact (smaller sample)
Incrementality Testing
For marketing or growth features, measure true incremental impact:
Method: Randomized controlled trial
Group A (90%): See the new growth feature
Group B (10%): Don't see it (control)
Measure: Difference in conversion between A and B
If A converts at 15% and B at 12%:
Incremental lift = 3 percentage points (25% relative)
6. Analytics Infrastructure
Event Tracking Schema
{
"event": "purchase_completed",
"timestamp": "2026-06-10T09:23:15Z",
"user_id": "usr_12345",
"session_id": "sess_67890",
"properties": {
"product_id": "prod_001",
"price": 49.99,
"currency": "USD",
"payment_method": "credit_card",
"coupon_applied": false
},
"context": {
"page_url": "/checkout",
"device": "mobile",
"os": "iOS 17",
"referrer": "email_campaign_summer"
}
}
Required events for every product:
| Event | When | Properties |
|---|
user_signed_up | Account creation | channel, referrer, device |
user_activated | First value moment | time_to_activate, feature_used |
session_started | App open / page load | referrer, device, os |
feature_used | Core feature interaction | feature_name, duration |
subscription_started | First payment | plan, amount, coupon |
subscription_cancelled | Churn | reason, tenure, LTV |
Tool Comparison
| Tool | Best For | Pricing |
|---|
| Amplitude | Product analytics, funnels, cohorts | Free tier generous |
| Mixpanel | Event tracking, user journeys | Free tier available |
| PostHog | Open-source, self-hosted option | Free + paid cloud |
| Heap | Auto-capture (no manual event setup) | Premium pricing |
| Google Analytics 4 | Web traffic, acquisition | Free |
| LaunchDarkly | Feature flags + experimentation | Paid |
| Statsig | Full-stack experimentation | Free tier available |
Dashboard Design
Executive Dashboard (weekly review):
┌─────────────────┬─────────────────┬─────────────────┐
│ NSM (WAU) │ Revenue (MRR) │ Retention D7 │
│ 45,200 │ $128,000 │ 28% │
│ ↑ 5% WoW │ ↑ 8% WoW │ → flat │
└─────────────────┴─────────────────┴─────────────────┘
┌─────────────────┬─────────────────┬─────────────────┐
│ Activation │ Conversion │ NPS Score │
│ 32% │ 4.2% │ 42 │
│ ↑ 2pp MoM │ ↑ 0.3pp │ ↑ 3 pts │
└─────────────────┴─────────────────┴─────────────────┘
Product Team Dashboard (daily):
- Funnel: Sign-up → Onboarding → Activation (last 7 days)
- Cohort: Week-1 retention by sign-up week
- Experiments: Active experiments + results
- Feature adoption: Top 10 features by usage
7. Common Analytics Pitfalls
| Pitfall | Why It Hurts | Fix |
|---|
| Vanity metrics | Downloads, page views don't correlate with value | Focus on NSM and retention |
| Survivorship bias | Only analyzing users who stayed | Include churned users in cohorts |
| Correlation = causation | Feature users retain more → feature causes retention? | Run experiments to prove causality |
| P-hacking | Testing many metrics until one is significant | Pre-register primary metric |
| Small sample sizes | Concluding from 100 users | Calculate sample size upfront |
| Ignoring seasonality | Comparing December to January | Use YoY or same-period comparisons |
| Segment-level effects | Overall metric flat but segment improving | Always segment by persona/channel |
Integration with Other Skills
| This skill provides | Related skill | For deeper dive |
|---|
| Metrics definition | business-product-leadership | JTBD → NSM mapping |
| Experiment design | business-product-leadership | A/B testing framework |
| Funnel/cohort analysis | business-product-leadership | Product health measurement |
| Statistical methods | research-design | Rigorous experimental protocol |
| Feature flags | collaborative-engineering-agent | Ship behind flags |
| Release tracking | diffusion-release-tracking | Gate advancement decisions |
| User research | product-ux-research | Qualitative insights |
