analysis

name	Analysis
description	Analyse the problem and come up with a hypothesis, write a snippet to test it, and measure the result.

Hypothesis-Driven Analysis & Model Debugging

You are doing empirical ML debugging. Follow this loop: hypothesize → snippet → measure → conclude → iterate.

---

The Loop

1. Form a hypothesis first

Before writing any code, state explicitly:

• What you think the problem is
• What metric/output would confirm or disprove it

2. Write a focused Python snippet

Keep snippets short and self-contained. Always print numbers — never just "it ran".

uv run python -c "
import pandas as pd, numpy as np
...
print('result:', round(value, 4))
"

Rules:

• One question per snippet
• Print the key number at the end
• If comparing two things, print both on the same line

3. Read the result and conclude

After seeing output, explicitly state:

• ✓ hypothesis confirmed / ✗ hypothesis rejected
• What this implies for the next step

4. Iterate or escalate

• If confirmed: implement the fix, measure again
• If rejected: form a new hypothesis
• If uncertain: design a more targeted test

---

Common Hypotheses to Check (ML)

CV leakage

• Are train/val splits leaking group information?
• Do samples in the same group (session, user, date) end up in both folds?
• Are any features constant within a group? (They become group IDs)

Distribution shift

• Are feature distributions significantly different between train and test?
• Use scipy.stats.ks_2samp — KS stat > 0.2 is a red flag
• Check if the shift is caused by temporal/contextual features

Feature causing memorization

• Remove the suspected feature set, re-measure CV score
• If CV drops but test score improves → the feature was a shortcut

Class imbalance effects

• Check AP per class, not just macro average
• Classes with <100 samples are at risk
• Check if rare classes are concentrated in few groups/dates

---

Snippet Patterns

Leave-one-group-out CV (honest when groups don't overlap with test):

for val_group in sorted(groups.unique()):
    mask = (groups == val_group).values
    m.fit(X[~mask], y[~mask])
    oof[mask] = m.predict_proba(X[mask])

KS test for distribution shift:

from scipy.stats import ks_2samp
for f in features:
    stat, _ = ks_2samp(train[f].dropna(), test[f].dropna())
    if stat > 0.2:
        print(f'{f}: KS={stat:.3f}  ← suspicious')

Per-class AP:

from sklearn.metrics import average_precision_score
y_bin = (y.to_numpy()[:, None] == classes).astype(int)
ap = average_precision_score(y_bin, probs, average=None)
for c, a in zip(classes, ap): print(f'  {c}: {a:.3f}')
print(f'macro: {ap.mean():.4f}')

Feature ablation:

for label, feats in [('all', all_feats), ('no_weather', no_weather), ('radar_only', radar_only)]:
    score = evaluate(feats)
    print(f'{label}: {score:.4f}')

---

Fixing the Problem

Once the root cause is confirmed:

1. Fix the CV first — a leaky CV masks everything else. Use the correct split strategy before tuning anything.
2. Fix features second — remove features that encode group/date identity.
3. Fix the model last — hyperparameter tuning on a leaky CV is wasted effort.

---

Output Format

After each experiment, summarize:

Hypothesis: [what you tested]
Result: [the number(s)]
Conclusion: [confirmed/rejected + implication]
Next: [what to try]

Before committing any change, verify the fix actually helps by re-running the honest CV.