corl-reproducibility

name	corl-reproducibility
description	Use when making a CoRL robot-learning paper reproducible — pinning simulator and driver versions, releasing training configs, demonstration data and checkpoints, documenting hardware setups that cannot be rerun, seed policy, evaluation scripts, and honest availability statements for code, data, and robot platforms.

CoRL Reproducibility

Robot-learning papers have a split reproducibility problem: the training half is software and can in principle be rerun anywhere, while the hardware half is a physical setup nobody else owns. Strong CoRL papers treat these halves differently — the software half is made rerunnable, the hardware half is made auditable — and say plainly which is which.

The rerunnable half: simulation and training

The silent reproducibility killers in this field are version-shaped:

• Simulator versions change physics. Contact solvers, default damping, and collision margins shift between releases of MuJoCo, Isaac, PyBullet, and friends; a policy's success rate is a function of the simulator build. Pin the exact version and any physics-relevant flags.
• Environment wrappers drift. Task-suite repositories (benchmark forks, custom reward shims) move under you; record the commit hash of every env repo, not just your own.
• GPU nondeterminism. cuDNN autotuning and atomics make bit-identical training runs unrealistic — so define reproducibility at the distribution level: same configs + fresh seeds should land inside your reported seed spread.

# repro-manifest.yaml — ship at the repo root; one block per results table
table_3:
  code_commit: ""            # your repo @ exact hash
  env_suite:  {repo: "", commit: "", sim: "mujoco==X.Y.Z", physics_flags: []}
  training:   {config: "configs/table3.yaml", seeds: [0,1,2,3,4], gpu: "1xA100-80GB", hours: 14}
  data:       {demos: "dataset-v2 (1,204 episodes)", url_or_status: "", license: ""}
  checkpoints: {released: true, path: "ckpts/table3/", selection_rule: "last epoch, no eval peeking"}
  evaluation: {script: "eval/run.py", episodes_per_task: 50, init_state_list: "eval/states.json"}
expected_tolerance: "per-task success within ±4 pts of Table 3 mean"

The selection_rule line matters more than it looks: checkpoint selection via test-set peeking is the field's quiet irreproducibility engine, and stating the rule is the cheapest credibility purchase available.

The auditable half: hardware

Nobody will re-run your robot, so the goal is that a skeptical expert could verify the experiment happened as described and rebuild an equivalent rig:

What to document	Why an auditor needs it
Robot model, end-effector, firmware/driver versions	Behavior differs across firmware, not just robots
Sensor models, mounting poses, calibration procedure	Camera extrinsics silently dominate visuomotor results
Control interface: frequency, action space, safety filters	"30 Hz end-effector deltas" vs "torque control" are different papers
Deployed compute + inference latency	Policy behavior is latency-dependent
Scene inventory: objects (make/size), fixtures, lighting	Enables an equivalent-rig rebuild and honest comparison
Raw episode logs and unedited evaluation video	The audit trail for every printed success rate

Log every evaluation episode at capture time (timestamped video plus a CSV of outcomes); retrofitting an audit trail after reviews ask for it is impossible.

Data and checkpoint release

• Demonstration datasets are results: release episode counts, collection method (teleop rig, scripted, crowdsourced), operator count, and filtering rules.
• Released checkpoints let others reproduce evaluation even when training is too expensive to repeat — for large policies this is often the highest-value artifact you can ship.
• If data or weights cannot be released (proprietary platform, human-subject footage), say so in the paper with the reason, and release what remains: configs, eval scripts, sim environments, and metrics logs. A precise partial-release statement outperforms a vague "code available upon request."

During review vs after acceptance

• During review everything must be anonymous (see corl-submission): anonymized repo mirrors, no lab-identifying video, no cloud buckets with named projects.
• The 2026 camera-ready pipeline has a sharp constraint: PMLR does not accept video as supplementary material, so post-acceptance videos, code, and data live on external hosting (project site, GitHub, archive), linked from the main text (corl.org author instructions, read 2026-07-08). Plan the public homes of artifacts before the October camera-ready deadline, and prefer DOI-stamped archives for anything you cite as permanent.
• No formal reproducibility checklist was verified for the 2026 cycle in this pack (待核实) — but reviewer expectations enforce one informally; use the manifest above regardless of what the form requires.

Availability statement patterns

Strong:  "Code, training configs, evaluation scripts, and the 1,204-episode
          teleop dataset: <URL>. Checkpoints for Tables 2-4: <URL>. Hardware
          evaluations are documented in Appendix C (rig spec, logs, uncut video);
          the platform itself cannot be redistributed."
Weak:    "Code will be released upon acceptance."      (unverifiable promise)
Broken:  "Results reproducible with standard settings." (no artifact at all)

Pre-submission audit

[ ] repro-manifest present; one block per headline table
[ ] Simulator/env/driver versions + commits pinned everywhere
[ ] Seed policy stated; checkpoint selection rule stated
[ ] Hardware rig spec complete enough for an equivalent rebuild
[ ] Episode-level logs + uncut eval video archived internally
[ ] Data/checkpoint release plan with named blockers, if any
[ ] Anonymous during review; external hosting plan ready for camera-ready

Verify the live cycle's supplementary and camera-ready rules at https://www.corl.org/contributions/instruction-for-authors before promising any artifact channel — PMLR-side constraints and CoRL-side forms both change yearly.