product-analytics

name	product-analytics
license	MIT
compatibility	Claude Code 2.1.220+.
description	A/B test evaluation, cohort retention analysis, funnel metrics, and experiment-driven product decisions. Use when analyzing experiments, measuring feature adoption, diagnosing conversion drop-offs, or evaluating statistical significance of product changes.
tags	["ab-test","cohort","retention","funnel","conversion","analytics","experiment","statistical-significance"]
context	fork
agent	product-strategist
version	1.0.0
author	OrchestKit
user-invocable	false
disable-model-invocation	false
complexity	medium
persuasion-type	guidance
metadata	{"category":"document-asset-creation"}
allowed-tools	["Read","Glob","Grep","WebFetch","WebSearch"]

Product Analytics

Frameworks for turning raw product data into ship/extend/kill decisions. Covers A/B testing, cohort retention, funnel analysis, and the statistical foundations needed to make those decisions with confidence.

Quick Reference

Category	Rules	Impact	When to Use
A/B Test Evaluation	1	HIGH	Comparing variants, measuring significance, shipping decisions
Cohort Retention	1	HIGH	Feature adoption curves, day-N retention, engagement scoring
Funnel Analysis	1	HIGH	Drop-off diagnosis, conversion optimization, stage mapping
Statistical Foundations	1	HIGH	p-value interpretation, sample sizing, confidence intervals

Total: 4 rules across 4 categories

A/B Test Evaluation

Load rules/ab-test-evaluation.md for the full framework. Quick pattern:

## Experiment: [Name]

Hypothesis: If we [change], then [primary metric] will [direction] by [amount]
  because [evidence or reasoning].

Sample size: [N per variant] — calculated for MDE=[X%], power=80%, alpha=0.05
Duration: [Minimum weeks] — never stop early (peeking bias)

Results:
  Control:   [metric value]  n=[count]
  Treatment: [metric value]  n=[count]
  Lift:      [+/- X%]        p=[value]  95% CI: [lower, upper]

Decision: SHIP / EXTEND / KILL
  Rationale: [One sentence grounded in numbers, not gut feel]

Decision rules:

• SHIP — p < 0.05, CI excludes zero, no guardrail regressions
• EXTEND — trending positive but underpowered (add runtime, not reanalysis)
• KILL — null result or guardrail degradation

See rules/ab-test-evaluation.md for sample size formulas, SRM checks, and pitfall list.

Cohort Retention

Load rules/cohort-retention.md for full methodology. Quick pattern:

-- Day-N retention cohort query
SELECT
  DATE_TRUNC(, first_seen)   cohort_week,
  ( user_id)          cohort_size,
  ( 
     activity_date  first_seen   
     user_id )  
     ( user_id)      day_7_retention
 user_activity
  
  ;