reproducibility-protocol

name	reproducibility-protocol
description	Ensure experiment reproducibility through systematic environment and seed control
version	1.0.0
category	experiment-execution
type	tactic
orchestrates	["seed-protocol-design","environment-specification"]
dependencies	{"sops":["environment-specification","seed-protocol-design"]}

Tactic: Reproducibility Protocol

Orchestration Pattern

1. Assess Reproducibility Requirements → Determine level needed (exact, statistical, conceptual)
2. seed-protocol-design → Design random seed strategy for all stochastic components
3. environment-specification → Lock all software, hardware, and configuration dependencies
4. Define Verification Plan → How to confirm reproducibility (re-run subset, cross-machine test)
5. Document Non-Determinism → Identify and document unavoidable sources of variance

Decision Criteria

Reproducibility Level	Requirement	When to Use
Exact (bit-for-bit)	Same hardware + deterministic ops	Debugging, verification
Statistical (within CI)	Same distribution of results	Standard research
Conceptual (same conclusion)	Same qualitative findings	Cross-platform validation

Quality Checks

• Are all random seeds documented and controllable?
• Is the full software environment captured (versions, dependencies)?
• Are hardware-specific non-determinisms identified (GPU atomics, cuDNN)?
• Is there a verification protocol (re-run N times, check variance)?
• Are data preprocessing steps deterministic or seeded?
• Is the order of data loading controlled?
• Are model initialization seeds separate from data shuffling seeds?
• Is there a plan for cross-machine reproducibility testing?

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

SOP	When to use
environment-specification	SOP: define complete experiment environment specification
seed-protocol-design	SOP: design random seed strategy for reproducibility