| name | corl-reproducibility |
| description | Use when making a CoRL robot-learning paper reproducible — pinning simulator and driver versions, releasing training configs, demonstration data and checkpoints, documenting hardware setups that cannot be rerun, seed policy, evaluation scripts, and honest availability statements for code, data, and robot platforms. |
CoRL Reproducibility
Robot-learning papers have a split reproducibility problem: the training half is
software and can in principle be rerun anywhere, while the hardware half is a
physical setup nobody else owns. Strong CoRL papers treat these halves
differently — the software half is made rerunnable, the hardware half is made
auditable — and say plainly which is which.
The rerunnable half: simulation and training
The silent reproducibility killers in this field are version-shaped:
- Simulator versions change physics. Contact solvers, default damping, and
collision margins shift between releases of MuJoCo, Isaac, PyBullet, and
friends; a policy's success rate is a function of the simulator build. Pin the
exact version and any physics-relevant flags.
- Environment wrappers drift. Task-suite repositories (benchmark forks,
custom reward shims) move under you; record the commit hash of every env repo,
not just your own.
- GPU nondeterminism. cuDNN autotuning and atomics make bit-identical
training runs unrealistic — so define reproducibility at the distribution
level: same configs + fresh seeds should land inside your reported seed spread.
table_3:
code_commit: ""
env_suite: {repo: "", commit: "", sim: "mujoco==X.Y.Z", physics_flags: []}
training: {config: "configs/table3.yaml", seeds: [0,1,2,3,4], gpu: "1xA100-80GB", hours: 14}
data: {demos: "dataset-v2 (1,204 episodes)", url_or_status: "", license: ""}
checkpoints: {released: true, path: "ckpts/table3/", selection_rule: "last epoch, no eval peeking"}
evaluation: {script: "eval/run.py", episodes_per_task: 50, init_state_list: "eval/states.json"}
expected_tolerance: "per-task success within ±4 pts of Table 3 mean"
The selection_rule line matters more than it looks: checkpoint selection via
test-set peeking is the field's quiet irreproducibility engine, and stating the
rule is the cheapest credibility purchase available.
The auditable half: hardware
Nobody will re-run your robot, so the goal is that a skeptical expert could
verify the experiment happened as described and rebuild an equivalent rig:
| What to document | Why an auditor needs it |
|---|
| Robot model, end-effector, firmware/driver versions | Behavior differs across firmware, not just robots |
| Sensor models, mounting poses, calibration procedure | Camera extrinsics silently dominate visuomotor results |
| Control interface: frequency, action space, safety filters | "30 Hz end-effector deltas" vs "torque control" are different papers |
| Deployed compute + inference latency | Policy behavior is latency-dependent |
| Scene inventory: objects (make/size), fixtures, lighting | Enables an equivalent-rig rebuild and honest comparison |
| Raw episode logs and unedited evaluation video | The audit trail for every printed success rate |
Log every evaluation episode at capture time (timestamped video plus a CSV of
outcomes); retrofitting an audit trail after reviews ask for it is impossible.
Data and checkpoint release
- Demonstration datasets are results: release episode counts, collection method
(teleop rig, scripted, crowdsourced), operator count, and filtering rules.
- Released checkpoints let others reproduce evaluation even when training is
too expensive to repeat — for large policies this is often the highest-value
artifact you can ship.
- If data or weights cannot be released (proprietary platform, human-subject
footage), say so in the paper with the reason, and release what remains:
configs, eval scripts, sim environments, and metrics logs. A precise
partial-release statement outperforms a vague "code available upon request."
During review vs after acceptance
- During review everything must be anonymous (see
corl-submission): anonymized
repo mirrors, no lab-identifying video, no cloud buckets with named projects.
- The 2026 camera-ready pipeline has a sharp constraint: PMLR does not accept
video as supplementary material, so post-acceptance videos, code, and data
live on external hosting (project site, GitHub, archive), linked from the main
text (corl.org author instructions, read 2026-07-08). Plan the public homes of
artifacts before the October camera-ready deadline, and prefer DOI-stamped
archives for anything you cite as permanent.
- No formal reproducibility checklist was verified for the 2026 cycle in this
pack (待核实) — but reviewer expectations enforce one informally; use the
manifest above regardless of what the form requires.
Availability statement patterns
Strong: "Code, training configs, evaluation scripts, and the 1,204-episode
teleop dataset: <URL>. Checkpoints for Tables 2-4: <URL>. Hardware
evaluations are documented in Appendix C (rig spec, logs, uncut video);
the platform itself cannot be redistributed."
Weak: "Code will be released upon acceptance." (unverifiable promise)
Broken: "Results reproducible with standard settings." (no artifact at all)
Pre-submission audit
[ ] repro-manifest present; one block per headline table
[ ] Simulator/env/driver versions + commits pinned everywhere
[ ] Seed policy stated; checkpoint selection rule stated
[ ] Hardware rig spec complete enough for an equivalent rebuild
[ ] Episode-level logs + uncut eval video archived internally
[ ] Data/checkpoint release plan with named blockers, if any
[ ] Anonymous during review; external hosting plan ready for camera-ready
Verify the live cycle's supplementary and camera-ready rules at
https://www.corl.org/contributions/instruction-for-authors before promising any
artifact channel — PMLR-side constraints and CoRL-side forms both change yearly.
