| name | srm-detection |
| description | Procedure for detecting and diagnosing Sample-Ratio Mismatch in A/B experiments — covers the chi-squared test, common causes by layer, and the decision rule for whether to trust or discard a result. |
SRM Detection
When to Use This
Before reading any metric from an A/B experiment. Sample-Ratio Mismatch (SRM) means the assignment split doesn't match the intended split — which means the groups are not comparable, and any observed difference may be assignment bias rather than treatment effect. Always check SRM before interpreting results.
What is SRM?
If an experiment is designed as 50/50, you expect roughly equal users in each variant. SRM occurs when the actual split deviates more than chance explains. A 50/50 experiment that actually lands 52/48 isn't necessarily SRM — with enough traffic, even a 51/49 can be statistically significant. The question is whether the deviation is beyond chance.
Step 1 — Run the Chi-Squared Test
Collect: n_control, n_treatment, and the intended split ratio (e.g., 0.5 / 0.5).
from scipy.stats import chisquare
n_control = 10432
n_treatment = 9811
total = n_control + n_treatment
intended_split = 0.5
expected_control = total * intended_split
expected_treatment = total * (1 - intended_split)
stat, p_value = chisquare(
f_obs=[n_control, n_treatment],
f_exp=[expected_control, expected_treatment]
)
print(f"Chi-squared: {stat:.2f}, p-value: {p_value:.4f}")
Decision rule:
p >= 0.01: no SRM detected — proceed to metric analysis.p < 0.01: SRM present — do not read metrics; investigate the cause first.
Use p < 0.01 (not 0.05) as the threshold — you're not testing a hypothesis about the product, you're testing the integrity of the apparatus. A more conservative threshold reduces false "all clear" verdicts.
Step 2 — Identify the Cause
SRM almost always comes from one of these layers:
| Layer | Common cause | Diagnostic |
|---|
| Assignment | Hash collision; non-uniform random; assignment bug in SDK | Check assignment distribution by time segment; look for all-one-variant periods |
| Exposure logging | Inconsistent exposure event firing; event dropped under load | Compare assignment count vs exposure event count per variant |
| Filtering | Post-assignment exclusion applied to only one variant (e.g., bot filter, geo filter) | Check exclusion counts per variant |
| Redirect / UX | One variant redirects to a different page, causing session drop | Compare bounce rates per variant in the first 30 seconds |
| Instrumentation | Tracking code only fires in treatment (e.g., new pixel added to variant) | Check event count per variant for session-start or page-load events |
| Deployment / rollout | Treatment wasn't fully deployed; some hosts serving old code | Check variant exposure by host/pod; look for partial rollout |
Step 3 — Segment the Check
If the overall chi-squared is borderline, run segmented checks:
for date, group in df.groupby('date'):
n_c = group[group.variant == 'control'].user_id.nunique()
n_t = group[group.variant == 'treatment'].user_id.nunique()
stat, p = chisquare([n_c, n_t], [n_c + n_t) * 0.5, (n_c + n_t) * 0.5])
print(f"{date}: n_c={n_c}, n_t={n_t}, p={p:.4f}")
A spike on a specific date points to a deployment event. A consistent imbalance from day one points to an assignment or exposure bug.
Decision Table
| SRM result | Action |
|---|
No SRM (p >= 0.01) | Proceed to metric analysis |
| SRM from known, correctable cause (e.g., one day of bad exposure logging) | Exclude the affected period; re-run chi-squared on clean data |
| SRM from assignment bug or hash collision | Stop experiment; fix the assignment mechanism; restart with clean assignment |
| SRM from filtering applied post-assignment | Redesign experiment to apply filter pre-assignment (only assign eligible users) |
| SRM cause unknown after investigation | Do not ship on this result; escalate to experimentation platform team |
Never ship a result with unexplained SRM. The metric difference may be real, but you cannot know which direction the bias runs.
Pitfalls
- Reading the primary metric before checking SRM — confirmation bias makes it harder to invalidate a significant result afterward.
- Using
p < 0.05 as the SRM threshold — too permissive; 5% false negatives on apparatus checks are unacceptable.
- Fixing SRM by trimming one variant to match the other's size — the trimmed users are not random; you've introduced selection bias.
- Checking SRM only at experiment end — check at 24 hours, 72 hours, and weekly to catch deployment-window bugs before the experiment runs to completion.
See Also
