| name | rss-reproducibility |
| description | Use when strengthening the reproducibility of an RSS (Robotics: Science and Systems) paper whose evidence lives partly on hardware — platform disclosure ledgers, log-backed trial claims, seeds and configs for the computational half, honest replication tiers, and release plans aligned with the free open-access proceedings culture. |
RSS Reproducibility
Make an embodied result checkable. Robotics reproducibility is two different problems
wearing one name: the computational half (code, seeds, configs) can be rerun by
anyone; the physical half (this robot, this room, this cloth) can only be specified
well enough for a peer lab to attempt replication. Treat the halves separately and
state which tier each claim sits in.
Replication tiers — declare one per claim
| Tier | What a reader can do | Minimum you must provide |
|---|
| Rerunnable | Reproduce the number from the release | Code, seeds, configs, data, one entry command |
| Re-collectable | Regenerate equivalent data on similar hardware | Full platform ledger + protocol document |
| Auditable | Verify your numbers follow from your logs | Raw trial logs + analysis scripts |
| Testimonial | Trust the authors | Nothing — and reviewers price it accordingly |
A paper mixing tiers is fine; a paper implying Rerunnable while delivering
Testimonial is the credibility failure RSS reviewers punish hardest.
The platform ledger
For the physical half, disclosure is the reproducibility. A peer lab needs:
- Robot make/model and any modifications; end-effector; sensor suite with mounting
and calibration procedure; control frequency and stack (firmware through policy).
- Environment specifics that carry load: table friction, lighting, object sourcing
(purchasable items beat lab-fabricated ones), workspace dimensions.
- Human procedure: reset steps, intervention rules, operator role — written as
instructions, not narrative.
- Timing facts that gate feasibility: wall-clock per trial, total campaign hours,
battery/thermal constraints that shaped the protocol.
Log-backed claims
The strongest robotics reproducibility artifact is the raw trial ledger:
trial_0142 cond=gripper_B obj=towel_3 seed=1142
result=fail cause=release_timing video=clips/0142.mp4
notes=operator_intervention:none
- One row per attempt, appended at collection time, never edited afterward.
- Analysis scripts consume the ledger and emit every table in the paper — so PDF
numbers cannot drift from evidence.
- Ship the ledger (anonymized) in the supplement; it converts "trust us" trial counts
into auditable ones.
What to capture during the campaign (not after)
Reproducibility in robotics is mostly a collection-time discipline; these cannot
be reconstructed later:
- Per-trial ledger rows (above) appended live, including aborted attempts.
- Camera footage of every trial, named by trial ID — storage is cheaper than a
re-run campaign, and it doubles as
rss-supplementary raw material.
- Calibration snapshots (camera extrinsics, force-torque zeroing) at session start,
so drift between sessions is diagnosable.
- Software state per session: commit hash, config checksum, simulator build.
- Environmental incidentals that later become reviewer questions: object wear,
lighting changes, floor surface swaps.
- A daily one-paragraph campaign log; six weeks later it is the only honest answer
to "why does condition B have 19 trials instead of 25?"
Computational half: standard, therefore mandatory
- Seeds for every stochastic component (policy training, sampler init, domain
randomization) and the variance across them, not one lucky seed.
- Exact dependency manifest, simulator version and physics timestep, GPU/CPU spec,
and training wall-clock.
- Config files as the single source of hyperparameters; no numbers living only in
the paper text.
Release plan and the venue's open culture
RSS proceedings are free to every reader at roboticsproceedings.org; a paywalled PDF
with closed evidence would be off-culture. Plan the release in two states: anonymized
supplement at submission (rss-supplementary), public repository + ledger + footage
at camera-ready (rss-camera-ready). If something cannot be released (proprietary
platform, safety), say what and why in the paper — a stated gap reads as honesty, a
silent one as concealment.
Vignette: an honest tier map
A paper couples a learned grasp ranker (trained in simulation) with hardware
clutter-clearing trials. Its declared map: the simulation training curves and
ranker metrics are Rerunnable (seeds, configs, dataset, one command); the
hardware success rates are Auditable (ledger plus analysis scripts shipped) and
Re-collectable for labs with a comparable arm (platform ledger and protocol
document provided); the wear-dependent behavior of one deformable object is
Testimonial, and the paper says so in one sentence. Reviewers can now disagree
with the evidence, but not discover its limits by surprise — which is the entire
game.
Simulator determinism caveats
- Physics engines are only conditionally deterministic: thread counts, contact
solver iterations, and hardware can change trajectories under the same seed.
Record engine version, timestep, solver settings — and state whether exact replay
or statistical equivalence is the reproduction target.
- GPU nondeterminism in training means "same seed" still needs a variance report
across seeds to be meaningful.
Output format
[Tier map] <claim -> Rerunnable/Re-collectable/Auditable/Testimonial>
[Platform ledger] complete / gaps: <list>
[Trial ledger] exists+shipped / exists / absent
[Computational gaps] <seeds/configs/versions/variance>
[Release plan] <submission state -> camera-ready state>
[Honesty debt] <implied tier above delivered tier, if any>
