academic-benchmarking

name	academic-benchmarking
description	Fair comparison methodology, controlling for hardware, citing published baselines, statistical significance in benchmarks. Use when designing experiments that compare systems or algorithms, or when reviewing benchmark results for publication.

Academic Benchmarking

Follow these principles when designing, running, or reporting benchmarks that compare systems, algorithms, or configurations in an academic context.

1. Fair Comparison Methodology

• Same Budget: Compare methods using the same computational budget (wall-clock time, number of function evaluations, or FLOPs) — not just "number of iterations," which can vary wildly in cost.
• Same Evaluation Protocol: All compared methods must use the identical evaluation pipeline, datasets, splits, and metrics. Any difference invalidates the comparison.
• Same Hardware: Run all methods on the same machine (or at minimum, equivalent hardware). Report the hardware specification alongside results.
• Same Software Stack: Pin library versions, compiler flags, and runtime settings. A benchmark run on PyTorch 2.0 is not directly comparable to one on PyTorch 1.13.

2. Baseline Selection

• Always include a trivial baseline: Random search, default configuration, or a simple heuristic. This establishes the floor and prevents over-claiming.
• Include the state-of-the-art: Cite and reproduce (or use official implementations of) the current best-known method. If you cannot reproduce it, state this explicitly and cite the published numbers.
• Re-run, don't just cite: Published numbers from other papers were collected under different conditions. Whenever possible, re-run baselines on your hardware with your evaluation protocol. If citing published numbers, clearly label them as "reported" vs. "reproduced."

3. Controlling for Randomness

• Run every method with multiple seeds (minimum 5, ideally 10+).
• Report mean ± standard deviation or median with interquartile range — never a single run.
• Use statistical significance tests (e.g., Wilcoxon signed-rank for paired comparisons, Mann-Whitney U for unpaired) and report p-values.
• Report effect sizes (e.g., Cohen's d, Cliff's delta) alongside p-values. A statistically significant but tiny improvement may not be practically meaningful.

4. Reporting Results

• Tables: Show mean ± std for each method on each benchmark. Bold the best result. Use underline or a different marker for second-best.
• Critical difference diagrams: For comparing many methods across many benchmarks, use Nemenyi post-hoc test with critical difference plots (see autorank or Orange libraries).
• Convergence curves: Show performance over time or iterations, not just final values. This reveals sample efficiency and convergence behavior.
• Ablation studies: When proposing a method with multiple components, show the contribution of each component by removing them one at a time.

5. Common Pitfalls to Avoid

• Cherry-picking benchmarks: Report results on all datasets you ran, not just the ones where your method wins.
• Overfitting to the test set: Never tune hyperparameters on the test set. Use a held-out validation set for tuning and report test set results exactly once.
• Ignoring wall-clock time: A method that achieves 1% better accuracy but takes 100× longer may not be a meaningful improvement. Report both quality and efficiency.
• Unfair hyperparameter tuning: If you extensively tune your method but use default hyperparameters for baselines, the comparison is unfair. Tune all methods with equivalent effort.

6. Reproducibility

• Release code, data, and configuration files needed to reproduce results.
• Include a single-command reproduction script (e.g., make reproduce or ./run_all.sh).
• Archive artifacts on a persistent platform (Zenodo, Figshare) with a DOI, not just a GitHub link.