| name | academic-benchmarking |
| description | Fair comparison methodology, controlling for hardware, citing published baselines, statistical significance in benchmarks. Use when designing experiments that compare systems or algorithms, or when reviewing benchmark results for publication. |
Academic Benchmarking
Follow these principles when designing, running, or reporting benchmarks that compare systems, algorithms, or configurations in an academic context.
1. Fair Comparison Methodology
- Same Budget: Compare methods using the same computational budget (wall-clock time, number of function evaluations, or FLOPs) — not just "number of iterations," which can vary wildly in cost.
- Same Evaluation Protocol: All compared methods must use the identical evaluation pipeline, datasets, splits, and metrics. Any difference invalidates the comparison.
- Same Hardware: Run all methods on the same machine (or at minimum, equivalent hardware). Report the hardware specification alongside results.
- Same Software Stack: Pin library versions, compiler flags, and runtime settings. A benchmark run on PyTorch 2.0 is not directly comparable to one on PyTorch 1.13.
2. Baseline Selection
- Always include a trivial baseline: Random search, default configuration, or a simple heuristic. This establishes the floor and prevents over-claiming.
- Include the state-of-the-art: Cite and reproduce (or use official implementations of) the current best-known method. If you cannot reproduce it, state this explicitly and cite the published numbers.
- Re-run, don't just cite: Published numbers from other papers were collected under different conditions. Whenever possible, re-run baselines on your hardware with your evaluation protocol. If citing published numbers, clearly label them as "reported" vs. "reproduced."
3. Controlling for Randomness
- Run every method with multiple seeds (minimum 5, ideally 10+).
- Report mean ± standard deviation or median with interquartile range — never a single run.
- Use statistical significance tests (e.g., Wilcoxon signed-rank for paired comparisons, Mann-Whitney U for unpaired) and report p-values.
- Report effect sizes (e.g., Cohen's d, Cliff's delta) alongside p-values. A statistically significant but tiny improvement may not be practically meaningful.
4. Reporting Results
- Tables: Show mean ± std for each method on each benchmark. Bold the best result. Use underline or a different marker for second-best.
- Critical difference diagrams: For comparing many methods across many benchmarks, use Nemenyi post-hoc test with critical difference plots (see
autorank or Orange libraries).
- Convergence curves: Show performance over time or iterations, not just final values. This reveals sample efficiency and convergence behavior.
- Ablation studies: When proposing a method with multiple components, show the contribution of each component by removing them one at a time.
5. Common Pitfalls to Avoid
- Cherry-picking benchmarks: Report results on all datasets you ran, not just the ones where your method wins.
- Overfitting to the test set: Never tune hyperparameters on the test set. Use a held-out validation set for tuning and report test set results exactly once.
- Ignoring wall-clock time: A method that achieves 1% better accuracy but takes 100× longer may not be a meaningful improvement. Report both quality and efficiency.
- Unfair hyperparameter tuning: If you extensively tune your method but use default hyperparameters for baselines, the comparison is unfair. Tune all methods with equivalent effort.
6. Reproducibility
- Release code, data, and configuration files needed to reproduce results.
- Include a single-command reproduction script (e.g.,
make reproduce or ./run_all.sh).
- Archive artifacts on a persistent platform (Zenodo, Figshare) with a DOI, not just a GitHub link.
