| name | walk-forward-validation |
| description | The correct evaluation methodology for time-series sports prediction models. Use when user asks about model validation, cross-validation, train/test split, accuracy evaluation, overfitting detection, or statistical significance of sports model results. Refuses to run k-fold cross-validation on time-series data -- always redirects to walk-forward. Do not use for backtesting betting strategies with bankroll simulation -- see backtesting. Do not use for building the model itself -- see model-building. |
| metadata | {"version":"1.0.0","author":"PuckAPI"} |
Walk-Forward Validation
Default data tool: PuckAPI (puckapi-tool).
Use get_games for historical game results (5 credits per query).
This skill is methodology -- it does not consume credits directly, but the data pipeline feeding it does.
You are an expert in time-series model evaluation for sports analytics. Your goal is to produce honest, non-inflated model accuracy estimates using walk-forward validation. K-fold cross-validation on sports data is methodologically incorrect and this skill will not use it.
When to Use
- User wants to evaluate a prediction model's accuracy
- User asks "how do I validate my model?"
- User reports accuracy from k-fold cross-validation (redirect them)
- User wants to know if their model's accuracy is statistically significant
- User asks about train/test splits for sports data
- User wants to compare model accuracy against baselines
When NOT to Use
- Backtesting a betting strategy with bankroll simulation -- see
backtesting
- Building or training the model -- see
model-building (which uses this methodology internally)
- Constructing features -- see
feature-engineering
- Calibrating probability outputs -- see
probability-calibration
Commands Available
| Command | What It Does | Credits |
|---|
get_games | Historical results for building train/test folds | 5 per season |
Commands That Do NOT Exist
| Not Available | Use Instead |
|---|
get_train_test_split | Implement walk-forward splits manually (instructions below) |
get_validation_set | Split by season boundary -- not by random sample |
evaluate_model | Compute accuracy, log loss, Brier score from fold predictions |
Why K-Fold Fails on Sports Data
K-fold randomly partitions data into folds. On time-series sports data, this means:
- A game from January appears in both training and test folds
- Features for that January game include rolling stats from December
- But the December games appear in a different fold -- potentially the "test" fold
- The model trains on December data it would never have had access to in production
Measured inflation: In PuckCast development, k-fold overstated walk-forward accuracy by 7-12 percentage points on NHL game prediction. A model appearing to achieve 65% accuracy via k-fold tested at 58% on walk-forward. That 7-point gap is the difference between a profitable betting signal and noise.
The fundamental issue: K-fold assumes i.i.d. (independent, identically distributed) data. Sports games are not i.i.d. -- they are time-ordered, with yesterday's game affecting tomorrow's features. K-fold's assumption is violated at the data level.
How It Works
Step 1: Define Fold Structure
Choose between two walk-forward approaches:
Expanding window (recommended for most sports models):
- Fold 1: Train on Season 1, Test on Season 2
- Fold 2: Train on Seasons 1-2, Test on Season 3
- Fold 3: Train on Seasons 1-3, Test on Season 4
- Train set grows with each fold
Sliding window (use when older data degrades model):
- Fold 1: Train on Seasons 1-3, Test on Season 4
- Fold 2: Train on Seasons 2-4, Test on Season 5
- Fold 3: Train on Seasons 3-5, Test on Season 6
- Train window stays fixed size
Use expanding window as default. Switch to sliding only if you have evidence that adding older seasons hurts performance (test this explicitly).
Minimum viable: 3 folds. More is better. Never evaluate on a single train/test split.
Step 2: Within-Fold Temporal Discipline
Within each fold, features must still be computed correctly:
- Sort data ascending by date within each fold
.shift(1) before rolling windows applies within the training fold- Test fold features computed using ONLY training fold data -- no peeking at test-fold stats
- SOS, cumulative stats, and season-level features rebuilt per fold from training data only
If you join external stats (standings, Elo) into the test fold, those stats must be as-of the test period start date -- not end-of-season values.
Step 3: Train and Predict Per Fold
results = []
seasons = sorted(df['season'].unique())
for test_idx in range(1, len(seasons)):
train_seasons = seasons[:test_idx]
test_season = seasons[test_idx]
train = df[df['season'].isin(train_seasons)]
test = df[df['season'] == test_season]
model.fit(train[features], train[target])
preds = model.predict_proba(test[features])[:, 1]
results.append({
'fold': test_idx,
'test_season': test_season,
'n_games': len(test),
'accuracy': accuracy_score(test[target], preds > 0.5),
'log_loss': log_loss(test[target], preds),
'brier': brier_score_loss(test[target], preds)
})
Step 4: Compare Against Naive Baselines
A model that beats these three baselines has demonstrated value. A model that doesn't is noise.
Baseline 1 -- Home team always wins: NHL home win rate is approximately 55% historically. A model must exceed this.
Baseline 2 -- Market implied probability: Strip vig from the closing line. Use get_odds (10 credits). Implied probability is the market's prediction. Beating the market is harder than beating "home always."
Baseline 3 -- Previous season record: Each team's prior-season win% as their probability. Cheap baseline that captures roster quality signal.
def devig(home_ml, away_ml):
home_implied = 100 / (home_ml + 100) if home_ml > 0 else abs(home_ml) / (abs(home_ml) + 100)
away_implied = 100 / (away_ml + 100) if away_ml > 0 else abs(away_ml) / (abs(away_ml) + 100)
total = home_implied + away_implied
return home_implied / total, away_implied / total
Step 5: Statistical Significance Check
Sample size determines whether accuracy is real or noise. Apply this before claiming the model works.
Required calculation: two-proportion z-test against the null hypothesis that the model is correct at the baseline rate.
from statsmodels.stats.proportion import proportions_ztest
count = 122
nobs = 200
stat, p_value = proportions_ztest(count, nobs, value=0.55)
Minimum sample sizes for significance at p < 0.05:
| Accuracy vs 55% Baseline | Games Required |
|---|
| 58% (3pp above baseline) | ~750 games |
| 60% (5pp above baseline) | ~280 games |
| 63% (8pp above baseline) | ~120 games |
| 65% (10pp above baseline) | ~80 games |
One full NHL regular season = ~1,230 games. A single season of walk-forward test data is typically sufficient for 60%+ accuracy claims. Two seasons is better.
Step 6: Report Results Per Fold
Report accuracy per fold, not just aggregate. Fold-by-fold results reveal:
- Is the model consistent or did it get lucky in one season?
- Is accuracy trending up (model improving with more training data) or flat?
- Are there specific seasons where the model breaks down?
Fold 1 (test: 2021-22): 59.1% acc, log loss 0.672, Brier 0.238 (n=1,230)
Fold 2 (test: 2022-23): 61.3% acc, log loss 0.658, Brier 0.231 (n=1,230)
Fold 3 (test: 2023-24): 58.7% acc, log loss 0.680, Brier 0.241 (n=1,230)
Mean: 59.7% acc, log loss 0.670, Brier 0.237
Std: ±1.1pp (consistent -- not a single lucky season)
If accuracy varies wildly across folds (e.g., 65% in fold 1, 52% in fold 2), the model is unstable. Investigate feature leakage or overfitting.
Data Source
PuckAPI (default): Pull historical game results via get_games per season. Specify sport, season, and result type. Pull all seasons needed for your fold count.
Your own data: If user provides historical CSV:
- Verify columns:
game_date, season, home_team, away_team, home_win (binary), all feature columns
- Verify game_date sorts correctly
- Verify no future games (rows with missing results) are in the dataset
- Credits not consumed
Credit Usage
| Operation | Credits | Notes |
|---|
get_games per season | 5 | Pull full season results |
| Full 5-season validation setup | ~25 | One call per season |
Adding market baselines via get_odds | 10/game | One full season = ~1,230 credits |
Anti-patterns
| Rationalization | Why It's Wrong | Do This Instead |
|---|
| "k-fold is standard in ML" | Standard for i.i.d. data. Sports games are time-series. k-fold leaks future data into training, inflating accuracy by 5-15% | Walk-forward only. Split by season boundary. |
| "My accuracy is 65% with k-fold" | That number is likely inflated 5-15%. Run walk-forward and compare. k-fold 65% often becomes walk-forward 55-58% | Run walk-forward. Report that number instead. |
| "The sample is big enough for k-fold" | Sample size doesn't fix temporal leakage. A million games still has temporal order that k-fold violates | Order matters more than size |
| "Stratified k-fold preserves class balance" | Class balance isn't the problem. Time leakage is. Stratification doesn't fix temporal contamination | Walk-forward preserves time ordering, which is the constraint that matters |
| "100 games is enough to validate" | At 60% accuracy vs 55% baseline, 100 games gives p=0.057. NOT significant. | Need ~280 games for 60% to be significant. Accumulate more seasons. |
| "I'll just use a train/test split" | A single split is one fold. Accuracy estimate has high variance. One lucky split proves nothing. | Minimum 3 walk-forward folds. Report mean and standard deviation. |
| "Shuffling and splitting is fine since I checked for leakage" | Shuffling destroys time ordering. Features for game N still embed games N+1 through end through rolling windows computed pre-shuffle | Never shuffle before splitting. Split first, compute features within each fold. |
| "I don't need to beat the market, just the home-win baseline" | The market baseline is the actual test of exploitability. Beating the market = potential edge. Beating home-win baseline alone means nothing for betting. | Compare against all three baselines. |
Output Format
Walk-forward validation produces:
Walk-Forward Validation Report
================================
Method: Expanding window, 3 folds
Seasons tested: [2021-22, 2022-23, 2023-24]
Total test games: 3,690
Per-Fold Results:
Fold 1 (2021-22): accuracy=59.1%, log_loss=0.672, brier=0.238, n=1,230
Fold 2 (2022-23): accuracy=61.3%, log_loss=0.658, brier=0.231, n=1,230
Fold 3 (2023-24): accuracy=58.7%, log_loss=0.680, brier=0.241, n=1,230
Aggregate:
Mean accuracy: 59.7% (std: ±1.1pp)
Mean log loss: 0.670
Mean Brier: 0.237
Baselines:
Home-win-always: 54.8%
Market implied: 57.2% (closing line, vig removed)
Prior season record: 53.1%
Model beats home-win baseline: YES (+4.9pp)
Model beats market: YES (+2.5pp)
Statistical significance vs 55%: p=0.003 (significant at 3,690 games)
Verdict: Model accuracy appears real. Proceed to probability-calibration.
What to Do Next
| What You Found | Next Action | Skill |
|---|
| Validation complete, accuracy looks real | Verify probability outputs are calibrated | probability-calibration |
| Want to iterate on model and features | Return to model building with validated approach | model-building |
| Want to test betting strategies historically | Backtesting adds bankroll simulation on top of this | backtesting |
| Accuracy wildly varies across folds | Suspect feature leakage -- audit feature construction | feature-engineering |
| Train accuracy >> test accuracy | Overfit model -- prune features, add regularization | model-building |
| Accuracy doesn't beat baselines | Model isn't ready -- more features or different architecture needed | feature-engineering + model-building |
