statistics

name	statistics
description	>-

Core Principles

1. Question first, method second — Define what you're trying to learn before selecting a test. The question determines the method, not the other way around.
2. Check assumptions before computing — Every test has assumptions (normality, independence, equal variance). Violating them silently produces misleading results.
3. Effect size matters more than p-values — A statistically significant result can be practically meaningless. Always report the magnitude of the effect alongside significance.
4. Confidence intervals over point estimates — A mean of 42 is less useful than "42 ± 5 (95% CI)". Always quantify uncertainty.
5. Don't test what you can describe — If the question is "what does the data look like?", use descriptive statistics. Hypothesis tests answer "is this difference real?", not "what is the data?"
6. One question, one test — Running multiple tests on the same data inflates false positives. If you must run multiple comparisons, correct for it (Bonferroni, Holm, or FDR).
7. Honest reporting — Report all results, not just significant ones. Pre-register your hypothesis when possible. State limitations clearly.

---

Method Selection Guide

Step 1: Classify Your Question

Question Type	What You're Asking	Go To
Describe	What does the data look like?	Descriptive Statistics
Compare	Are these groups different?	Comparison Tests
Relate	Do these variables move together?	Correlation & Regression
Predict	What will happen next?	Regression & Modeling (beyond this skill — use ML)
Classify	Does this differ from expected?	Goodness-of-Fit Tests

Step 2: Identify Your Data Structure

Factor	Options	Why It Matters
Number of groups	1, 2, or 3+	Determines which test family to use
Paired or independent?	Same subjects measured twice vs. different subjects	Paired tests are more powerful but require matched data
Variable type	Continuous (interval/ratio) vs. categorical (nominal/ordinal)	Determines parametric vs. non-parametric
Sample size	Small (< 30) vs. large (≥ 30)	Small samples need non-parametric or exact tests
Distribution shape	Normal vs. skewed vs. unknown	Parametric tests assume normality

---

Descriptive Statistics

Use these to summarize and understand data before testing anything.

Central Tendency

Measure	When to Use	Sensitive To
Mean	Symmetric, roughly normal data	Outliers (a single extreme value shifts it)
Median	Skewed data or when outliers are present	Nothing — robust by design
Mode	Categorical data or multimodal distributions	Ties, small samples

Rule: If mean ≠ median by more than 10–15%, the data is skewed — prefer median.

Spread

Measure	When to Use	Notes
Standard deviation	Normal-ish data; pair with mean	Same units as data
IQR (Q3 − Q1)	Skewed data; pair with median	Robust to outliers
Range	Quick sense of spread	Useless with outliers
Coefficient of variation (CV)	Comparing spread across datasets with different scales	CV = std / mean

Distribution Shape

Check	Method	Interpretation
Normality	Shapiro-Wilk (n < 5000), Anderson-Darling, or Q-Q plot	p < 0.05 → not normal
Skewness	scipy.stats.skew	|skew| > 1 = substantially skewed
Kurtosis	scipy.stats.kurtosis	> 0 = heavy tails, < 0 = light tails

---

Comparison Tests

Choosing the Right Test

How many groups?
├── 1 group vs. known value
│   ├── Normal → One-sample t-test
│   └── Not normal → Wilcoxon signed-rank
├── 2 groups
│   ├── Paired (same subjects)?
│   │   ├── Normal → Paired t-test
│   │   └── Not normal → Wilcoxon signed-rank
│   └── Independent?
│       ├── Normal, equal variance → Independent t-test
│       ├── Normal, unequal variance → Welch's t-test
│       └── Not normal → Mann-Whitney U
└── 3+ groups
    ├── Normal, equal variance → One-way ANOVA
    ├── Not normal → Kruskal-Wallis
    └── Repeated measures → Repeated measures ANOVA or Friedman test

Test Reference

Test	Compares	Assumptions	Non-Parametric Alternative
One-sample t-test	Sample mean vs. known value	Normal, continuous	Wilcoxon signed-rank
Independent t-test	Means of 2 independent groups	Normal, equal variance, independent	Mann-Whitney U
Welch's t-test	Means of 2 independent groups	Normal, independent (no equal variance needed)	Mann-Whitney U
Paired t-test	Means of 2 paired measurements	Normal differences, continuous	Wilcoxon signed-rank
One-way ANOVA	Means of 3+ independent groups	Normal, equal variance, independent	Kruskal-Wallis
Chi-squared test	Observed vs. expected frequencies	Expected count ≥ 5 in each cell	Fisher's exact test

Post-Hoc Tests (After ANOVA / Kruskal-Wallis)

If the omnibus test is significant (groups differ), use post-hoc tests to find which groups differ:

Test	When to Use
Tukey HSD	All pairwise comparisons, equal sample sizes
Bonferroni correction	Few planned comparisons
Dunn's test	Post-hoc for Kruskal-Wallis

---

Correlation and Association

Method	Variable Types	Measures	Assumptions
Pearson's r	Both continuous	Linear relationship strength	Normal, linear, no outliers
Spearman's ρ	Ordinal or continuous	Monotonic relationship strength	None (rank-based)
Kendall's τ	Ordinal, small samples	Monotonic relationship strength	None (rank-based, more robust than Spearman for small n)
Chi-squared test of independence	Both categorical	Whether variables are associated	Expected counts ≥ 5
Point-biserial	One binary, one continuous	Correlation	Normal continuous variable

Correlation ≠ causation. Always state this explicitly when reporting. A correlation is a starting point for investigation, not a conclusion.

---

Effect Size

Always report effect size alongside significance. A tiny p-value with a tiny effect is not actionable.

Context	Measure	Small	Medium	Large
2-group comparison	Cohen's d	0.2	0.5	0.8
ANOVA	Eta-squared (η²)	0.01	0.06	0.14
Correlation	r or R²	0.1	0.3	0.5
Chi-squared	Cramér's V	0.1	0.3	0.5
Binary outcome	Odds ratio	1.5	2.5	4.0

---

Sample Size and Power

Before running an experiment, calculate the required sample size:

## Power Analysis

**Test**: <e.g., "Independent t-test">
**Desired power**: 0.80 (standard) or 0.90 (high confidence)
**Significance level (α)**: 0.05
**Minimum detectable effect (MDE)**: <e.g., "Cohen's d = 0.3" or "5% conversion lift">
**Required sample size per group**: <computed value>
**Expected duration to collect**: <e.g., "2 weeks at current traffic">

Rules of thumb:

• Power < 0.80 = underpowered; you'll likely miss real effects
• Larger effects need smaller samples; smaller effects need larger samples
• When in doubt, aim for more data — underpowered studies waste everyone's time

---

A/B Testing

A/B tests are the most common applied statistics scenario. Follow this workflow:

Before the Test

1. State the hypothesis: "Variant B will increase conversion rate by ≥ 5% relative to control"
2. Choose the metric: Primary metric (what you're optimizing) + guardrail metrics (what must not degrade)
3. Run power analysis: Calculate required sample size per variant
4. Set the duration: Don't peek at results before the required sample size is reached
5. Pre-register: Document hypothesis, metric, sample size, and success criteria before launching

During the Test

• Don't peek: Checking results daily and stopping early inflates false positives
• Don't change the test mid-flight: Adding variants, changing allocation, or modifying the feature during the test invalidates results
• Monitor guardrails: If error rates or latency spike, you have a quality issue, not a test result

After the Test

## A/B Test Results

**Test**: <name>
**Duration**: <start – end>
**Sample size**: Control: <n>, Variant: <n>

**Primary metric**: <metric name>
| Group | Value | 95% CI |
|-------|-------|--------|
| Control | <X> | [<lo>, <hi>] |
| Variant | <X> | [<lo>, <hi>] |

**Relative lift**: <X%> [<CI lo>, <CI hi>]
**p-value**: <X>
**Effect size**: <measure and value>

**Guardrail metrics**: <all stable / degradation in X>

**Decision**: <Ship variant | Keep control | Inconclusive — extend or redesign>
**Reasoning**: <why this decision, referencing effect size and practical significance>

---

Assumption Validation Checklist

Before applying any parametric test, verify:

Assumption	How to Check	If Violated
Normality	Shapiro-Wilk test, Q-Q plot, histogram	Use non-parametric alternative
Equal variance	Levene's test, F-test	Use Welch's t-test or rank-based test
Independence	Study design (not a statistical test)	Use paired/repeated measures test
Linearity (regression)	Scatter plot of residuals vs. fitted	Transform variables or use non-linear model
No multicollinearity (regression)	VIF < 5 for each predictor	Remove or combine correlated predictors

---

Process

Step 1: State the Question

## Statistical Question

**Question**: <What are you trying to learn? Be specific.>
**Population**: <Who or what does this apply to?>
**Hypothesis** (if testing): 
  - H₀: <null hypothesis — no effect, no difference>
  - H₁: <alternative hypothesis — the effect you expect>
**Practical significance threshold**: <What size effect would actually matter?>

Step 2: Profile the Data

## Data Profile

**Source**: <where the data comes from>
**Sample size**: <n>
**Variables**: 
  - <var1>: <type> — <description>
  - <var2>: <type> — <description>
**Missing data**: <count, percentage, pattern>
**Distribution**: <per variable — normal, skewed, categorical frequencies>
**Assumption checks**: <normality, variance, independence — pass/fail>

Step 3: Select the Method

Use the selection guide above. Document:

## Method Selection

**Test**: <name>
**Why**: <reasoning tied to question type, data structure, and assumption checks>
**Alternative considered**: <what else could work and why it wasn't chosen>
**Correction applied**: <e.g., "Bonferroni for 3 pairwise comparisons" or "none — single test">

Step 4: Compute and Interpret

## Results

**Test statistic**: <name> = <value>
**p-value**: <value>
**Effect size**: <measure> = <value> (<small | medium | large>)
**Confidence interval**: [<lower>, <upper>] at <confidence level>

**Interpretation**: <Plain-language statement of what this means. Not just "p < 0.05 so we reject H₀" — state the practical implication.>

**Limitations**: <What this result does NOT tell us. Confounders, generalizability, assumptions that were borderline.>

---

Output Format

Every statistical analysis should produce:

1. Question — What you're trying to learn, with hypothesis if applicable
2. Data Profile — Summary of the data, assumption checks
3. Method Selection — Which test, why, and what alternatives were considered
4. Results — Test statistic, p-value, effect size, confidence interval
5. Interpretation — Plain-language meaning and practical significance
6. Limitations — What the result doesn't tell you

---

Regression Basics

Regression is the most common applied statistics method. Use this decision guide:

Outcome Variable	Predictor(s)	Method	Use When
Continuous	1 continuous	Simple linear regression	Predicting one variable from another (e.g., revenue from ad spend)
Continuous	Multiple	Multiple linear regression	Predicting with several factors; controlling for confounders
Binary (0/1)	Any	Logistic regression	Predicting yes/no outcomes (e.g., churn, conversion)
Count (0, 1, 2...)	Any	Poisson regression	Predicting event counts (e.g., support tickets per day)

Before running regression, check:

• Linearity (scatter plot of residuals vs. fitted)
• Independence of residuals
• Homoscedasticity (constant variance of residuals)
• No multicollinearity (VIF < 5 for each predictor)
• Normality of residuals (for inference, not prediction)

Key outputs to report: R², adjusted R², coefficients with CIs, residual plots, and F-test p-value.

---

When to Consider Bayesian Methods

Frequentist methods (everything above) are the default. Consider Bayesian approaches when:

Signal	Why Bayesian Helps
Small sample size (n < 30)	Priors regularize unstable estimates
You have strong prior knowledge	Incorporating domain expertise improves estimates
You need P(hypothesis | data), not P(data | hypothesis)	Credible intervals answer "what's the probability the effect is > X?" directly
Sequential testing / continuous monitoring	Bayesian A/B tests allow peeking without inflating error rates
Stakeholders struggle with p-values	"95% probability the effect is between 2% and 8%" is more intuitive

Practical tools: pymc, arviz for general Bayesian analysis; Bayesian A/B testing via bayesian-testing or built-in platform features (Optimizely, LaunchDarkly).

Default stance: Use frequentist methods unless one of the above signals is present. Don't switch to Bayesian for complexity's sake.

---

Tooling Reference

Library	Language	Best For
scipy.stats	Python	Hypothesis tests, distributions, descriptive stats
statsmodels	Python	Regression, ANOVA, time series, assumption diagnostics
pingouin	Python	Clean API for t-tests, ANOVA, correlation, effect sizes
scikit-learn	Python	Train/test splits, cross-validation, preprocessing
pymc	Python	Bayesian modeling and inference
power_analysis / statsmodels.stats.power	Python	Sample size and power calculations

---

Common Pitfalls

Pitfall	Why It Fails
p-hacking	Running multiple tests and reporting only significant results inflates false positives. Pre-register your hypothesis.
Ignoring effect size	p = 0.001 with Cohen's d = 0.05 means "we're very sure about a trivially small difference." Not actionable.
Small sample, big claims	A study with n = 12 that finds p = 0.04 is fragile. One outlier changes the conclusion.
Violating independence	Using the same users in both groups, or multiple measurements without paired tests. Results are invalid.
Peeking at A/B tests	Checking daily and stopping when p < 0.05 dramatically inflates false positive rate. Use sequential testing if you must peek.
Treating non-significant as "no effect"	Absence of evidence ≠ evidence of absence. You may be underpowered. Report power.
Applying parametric tests to ordinal data	A Likert scale (1–5) is not continuous. Use non-parametric methods.
Confusing correlation with causation	Pearson r = 0.8 does not mean X causes Y. It means they move together.
Cherry-picking subgroups	"It wasn't significant overall, but it was significant for users aged 25–30 on Tuesdays." This is noise.
Reporting without uncertainty	"Conversion rate is 4.2%" is less useful than "4.2% ± 0.8% (95% CI)." Always show the interval.