ml-benchmark-evaluation

name	ml-benchmark-evaluation
description	Rigorous methodology for evaluating ML models on established benchmarks. Covers proper train/val/test splits, baseline verification from original papers, exact metric formula discrepancies, data-leak detection checklist, multi-seed robustness, and honest reporting templates. Use when claiming to beat published baselines, writing methods papers, or auditing existing results.
category	ml-training
version	1.0.0
author	Synthetic Sciences
license	MIT
tags	["Evaluation","Benchmark","Validation","Train-Val-Test","Metrics","Data Leak","Reproducibility"]
dependencies	["torch","numpy"]

ML Benchmark Evaluation — Rigorous Methodology

When to Use

• Claiming to beat published baselines on any ML benchmark
• Writing a methods paper that compares against prior work
• Any ML evaluation where the reported number matters for publication
• Auditing existing results for overfitting or data leakage

The Golden Rule

Your evaluation protocol must be AT LEAST as rigorous as the baseline you claim to beat. Ideally, stricter.

1. Train/Val/Test Split

The Wrong Way (common but weak):

train = data[:9000]   # 90%
test = data[9000:]    # 10% — used for BOTH model selection AND final metric
# Problem: model selection on test set inflates reported performance

The Right Way:

N_TRAIN, N_VAL, N_TEST = 8000, 1000, 1000
train = data[:N_TRAIN]                          # Gradient updates
val = data[N_TRAIN:N_TRAIN+N_VAL]               # Model selection (checkpoints)
test = data[N_TRAIN+N_VAL:N_TRAIN+N_VAL+N_TEST] # Final metric (ONCE)

# During training:
if epoch % 50 == 0:
    val_metric = evaluate(model, val_loader)  # NOT test_loader
    if val_metric < best_val_metric:
        save_checkpoint(model)

# After training (ONCE):
model = load_best_checkpoint()
final_metric = evaluate(model, test_loader)  # This is the reported number

When the Benchmark Doesn't Use Val Split:

Many benchmarks (PDEBench, some Kaggle, older CV benchmarks) don't use validation splits. When claiming to beat them:

1. Report results using their protocol for fair comparison
2. ALSO report results using proper val split for scientific rigor
3. Be transparent about both numbers in the paper

2. Verify Published Baselines (NEVER Trust Task Descriptions)

Published numbers can be wrong in secondary sources. Always verify from the original paper.