statistical-analysis-when-to-use

name	statistical-analysis-when-to-use
description	Sub-skill of statistical-analysis: When to Use (+4).
version	1.0.0
category	data-analytics
type	reference
scripts_exempt	true

When to Use (+4)

When to Use

Use hypothesis testing when you need to determine whether an observed difference is likely real or could be due to random chance. Common scenarios:

• A/B test results: Is variant B actually better than A?
• Before/after comparison: Did the product change actually move the metric?
• Segment comparison: Do enterprise customers really have higher retention?

The Framework

1. Null hypothesis (H0): There is no difference (the default assumption)
2. Alternative hypothesis (H1): There is a difference
3. Choose significance level (alpha): Typically 0.05 (5% chance of false positive)
4. Compute test statistic and p-value
5. Interpret: If p < alpha, reject H0 (evidence of a real difference)

Common Tests

Scenario	Test	When to Use
Compare two group means	t-test (independent)	Normal data, two groups
Compare two group proportions	z-test for proportions	Conversion rates, binary outcomes
Compare paired measurements	Paired t-test	Before/after on same entities
Compare 3+ group means	ANOVA	Multiple segments or variants
Non-normal data, two groups	Mann-Whitney U test	Skewed metrics, ordinal data
Association between categories	Chi-squared test	Two categorical variables

Practical Significance vs. Statistical Significance

Statistical significance means the difference is unlikely due to chance.

Practical significance means the difference is large enough to matter for business decisions.

A difference can be statistically significant but practically meaningless (common with large samples). Always report:

• Effect size: How big is the difference? (e.g., "Variant B improved conversion by 0.3 percentage points")
• Confidence interval: What's the range of plausible true effects?
• Business impact: What does this translate to in revenue, users, or other business terms?

Sample Size Considerations

• Small samples produce unreliable results, even with significant p-values
• Rule of thumb for proportions: Need at least 30 events per group for basic reliability
• For detecting small effects (e.g., 1% conversion rate change), you may need thousands of observations per group
• If your sample is small, say so: "With only 200 observations per group, we have limited power to detect effects smaller than X%"