reproducibility-checklist

name	reproducibility-checklist
description	ML reproducibility checklist, hardware specs, data versioning, code archival. Use prior to publishing code, sending a paper for review, or finalizing an experiment setup.

Reproducibility Checklist

Prior to finalizing a paper, experiment suite, or publishing an open-source repository based on research, adhere to this checklist inspired by the ML Reproducibility Checklist (Pineau et al.).

1. For Models and Algorithms

• Description: Are all hyperparameters, including those not tuned, explicitly documented?
• Bounds: Is the exact search space for all hyperparameters defined?
• Metrics: Are the evaluation metrics mathematically defined (or standard implementations cited)?
• Baselines: Are baselines clearly described, including whether they were re-implemented or run from official sources?

2. For Datasets and Workloads

• Provenance: Are the datasets/workloads publicly available? If not, is a synthetic equivalent provided?
• Splits: Are the exact train/validation/test splits documented (or the seed used to generate them)?
• Preprocessing: Are all data filtering, normalization, or preprocessing steps explicitly scripted?

3. For Experimental Setup

• Hardware Specs: Is the full hardware description included? (CPU architecture, core count, RAM, Disk type (NVMe/SSD/HDD), Network conditions).
• Software Environment: Are OS, kernel version, runtime versions (e.g., Python 3.11, CUDA 12.1), and framework versions listed?
• Dependencies: Is a locked requirements.txt, Pipfile.lock, or Dockerfile provided?
• Randomness: Are the exact random seeds used reported?
• Statistical Significance: Are error bars, confidence intervals, or standard deviations reported alongside central tendencies?
• Budget: Is the total computational budget (e.g., GPU hours, wall-clock time) reported?

4. For Source Code

• Readme: Does an entry point README.md exist containing clear instructions on how to install dependencies and run a minimal example?
• Automation: Is there a single script (e.g., run_all.sh or a Makefile) that reproduces the key results or tables in the paper?
• Pre-trained Models/Configs: Are final generated configurations, models, or data artifacts accessible via an open repository (e.g., Zenodo, Figshare)?