Rigorous methodology for evaluating ML models on established benchmarks. Covers proper train/val/test splits, baseline verification from original papers, exact metric formula discrepancies, data-leak detection checklist, multi-seed robustness, and honest reporting templates. Use when claiming to beat published baselines, writing methods papers, or auditing existing results.
Rigorous methodology for evaluating ML models on established benchmarks. Covers proper train/val/test splits, baseline verification from original papers, exact metric formula discrepancies, data-leak detection checklist, multi-seed robustness, and honest reporting templates. Use when claiming to beat published baselines, writing methods papers, or auditing existing results.
Claiming to beat published baselines on any ML benchmark
Writing a methods paper that compares against prior work
Any ML evaluation where the reported number matters for publication
Auditing existing results for overfitting or data leakage
The Golden Rule
Your evaluation protocol must be AT LEAST as rigorous as the baseline you claim to beat. Ideally, stricter.
1. Train/Val/Test Split
The Wrong Way (common but weak):
train = data[:9000] # 90%
test = data[9000:] # 10% — used for BOTH model selection AND final metric# Problem: model selection on test set inflates reported performance
The Right Way:
N_TRAIN, N_VAL, N_TEST = 8000, 1000, 1000
train = data[:N_TRAIN] # Gradient updates
val = data[N_TRAIN:N_TRAIN+N_VAL] # Model selection (checkpoints)
test = data[N_TRAIN+N_VAL:N_TRAIN+N_VAL+N_TEST] # Final metric (ONCE)# During training:if epoch % 50 == 0:
val_metric = evaluate(model, val_loader) # NOT test_loaderif val_metric < best_val_metric:
save_checkpoint(model)
# After training (ONCE):
model = load_best_checkpoint()
final_metric = evaluate(model, test_loader) # This is the reported number
When the Benchmark Doesn't Use Val Split:
Many benchmarks (PDEBench, some Kaggle, older CV benchmarks) don't use validation splits. When claiming to beat them:
Report results using their protocol for fair comparison
ALSO report results using proper val split for scientific rigor
Be transparent about both numbers in the paper
2. Verify Published Baselines (NEVER Trust Task Descriptions)
Published numbers can be wrong in secondary sources. Always verify from the original paper.
# Download and parse the original paper
wget -O paper.pdf "https://arxiv.org/pdf/XXXX.XXXXX"
npm i -g @llamaindex/liteparse
liteparse parse paper.pdf -o paper_parsed.md
grep "nRMSE\|accuracy\|F1" paper_parsed.md
Common discrepancies found in practice:
Task specification says 5.9e-3 but paper says 9.7e-3 (wrong table, wrong metric)
Baseline from a different parameter setting or model configuration
RMSE vs nRMSE vs MSE confusion
Different train/test split than claimed
3. Data Leak Detection Checklist
Run these 6 checks before reporting any result:
# Check 1: IC window preserved exactly (for time-series / PDE problems)assert np.allclose(preds[:,:,:INIT_STEP], targets[:,:,:INIT_STEP], atol=1e-10)
# Check 2: Predictions differ from targets in predicted windowassertnot np.allclose(preds[:,:,INIT_STEP:], targets[:,:,INIT_STEP:], atol=1e-6)
# Check 3: No duplicate samples between train and test
train_hashes = set(hash(x.tobytes()) for x in train_data)
test_hashes = set(hash(x.tobytes()) for x in test_data)
assertlen(train_hashes & test_hashes) == 0# Check 4: Test set never used for gradients# Verify by code inspection: test_loader only in torch.no_grad() blocks# Check 5: No NaN/Inf in predictionsassertnot np.isnan(preds).any() andnot np.isinf(preds).any()
# Check 6: Error distribution is realistic# If min error is near-zero for many samples, suspiciousassert (per_sample_error < 1e-6).mean() < 0.01# <1% near-perfect
4. Metric Computation (Get It Exactly Right)
Different benchmarks use different metrics. Verify the EXACT formula from the benchmark code.
The SAME metric name (e.g., "nRMSE") can have multiple valid definitions that give different numbers (up to 5-10% difference):
# Formula A: Per-timestep, then average (common in code implementations)
err_per_t = sqrt(mean_spatial((pred-target)^2)) # [N, C, T]
nrm_per_t = sqrt(mean_spatial(target^2))
nrmse_A = mean(err_per_t / nrm_per_t) # average over N, C, T# Formula B: Frobenius norm ratio per sample (canonical PDEBench definition)
nrmse_B = mean_over_N(||pred_i - target_i||_F / ||target_i||_F)
# Formula C: Global RMSE / global RMS
nrmse_C = sqrt(mean_all((pred-target)^2)) / sqrt(mean_all(target^2))
These are NOT equivalent. For a single benchmark comparison, the difference can be 1-10%. Always:
Read the benchmark's actual metrics.py code (not just the paper)
Compute your metric using the EXACT same formula
If in doubt, report BOTH formulas and show you beat under both
Document which formula you used in results.json
5. Multi-Seed Robustness
Single-seed results can be lucky. For strong claims:
seeds = [42, 123, 7, 2024, 31415]
results = []
for seed in seeds:
torch.manual_seed(seed)
model = train(seed)
results.append(evaluate(model))
mean = np.mean(results)
std = np.std(results)
print(f"nRMSE = {mean:.4e} ± {std:.4e} (n={len(seeds)} seeds)")
Minimum for publication: 3 seeds for key results, report mean ± std.
6. Honest Reporting Template
## Results
| Test | Our nRMSE | Published | Improvement | Seeds | Val Split |
|------|-----------|-----------|-------------|-------|-----------|
| A | X.Xe-Y ± Z.Ze-Y | P.Pe-Q | N.N× | 3 | Yes (8K/1K/1K) |
### Comparison Fairness Notes:- Our model uses [more modes / higher resolution / ...] than the baseline
- These confounding factors are documented in Table X
- Ablation study (Table Y) isolates the contribution of each change
### Limitations:- [List every weakness honestly]
