| name | icra-artifact-evaluation |
| description | Use when packaging the artifacts behind an ICRA paper — ROS packages, controllers, simulation environments, trained policies, CAD and PCB files, datasets, and trial logs — into something a robotics reviewer or reader can actually run or audit, given that ICRA has no formal artifact-badging track to certify it for you. |
ICRA Artifact Evaluation
ICRA, unlike several software-systems conferences, has had no standing artifact
evaluation committee or badge system in recent cycles (re-check the current
year's calls before asserting this to authors — tracks appear and disappear).
The absence cuts both ways: nobody will certify your artifact, and nobody will
catch its problems before readers do. This skill applies an artifact-evaluator's
discipline voluntarily, because in robotics the artifact often is the
contribution's proof.
The robotics artifact stack
A robotics paper's artifact is rarely just "the code." Inventory all six layers
and decide, per layer, released / partially released / withheld-with-reason:
| Layer | Typical contents | Common blocker |
|---|
| Algorithms | planners, controllers, learning code | none — release |
| Integration | ROS launch files, configs, calibration | "works only on our stack" |
| Simulation | worlds, robot models (URDF/SDF), randomization | third-party asset licenses |
| Learned weights | trained policies, perception models | training data licensing |
| Hardware design | CAD, PCB, BOM for custom parts | patent/commercialization plans |
| Evidence | rosbags, trial videos, session sheets | raw-log size (TB-scale) |
A paper whose novelty is a custom end effector but which releases only Python
scripts has released the wrong layer. Match the release to the claim.
The five-minute-skeptic standard
Package for a reviewer who gives you five minutes before forming a judgment:
git clone <anonymized-artifact-url> && cd artifact
docker build -t icra-artifact .
docker run icra-artifact make figure3
docker run icra-artifact make table2
cat CLAIMS.md
- The first runnable thing must not require a robot: a simulation reproduction
or a log-replay analysis gives the skeptic a win in minutes.
CLAIMS.md maps each paper claim to a command and its expected output, and
explicitly lists which claims require physical hardware (audit tier — see
icra-reproducibility).- Pin everything: base image, ROS distro, Python deps, simulator version.
"Latest Gazebo" is a bug report generator.
Hardware-dependent artifacts
For layers that need the physical robot:
- Ship a log-replay mode: the perception and decision stack runs against
released rosbags, letting readers verify the pipeline without the arm.
- Ship the sim twin: the same launch files targeting the simulated platform,
clearly marked as not the source of the paper's hardware numbers.
- Document the hardware interface narrowly (which driver topics/services the
stack expects) so ports to other platforms are feasible.
- For custom mechanisms, releasing CAD + BOM converts "trust our gripper" into
"build our gripper"; if commercialization blocks this, say so in the paper
rather than staying silent.
Datasets and logs as artifacts
- Extracted per-trial features plus a representative raw sample beat an
undifferentiated terabyte dump; provide a download script with checksums.
- License explicitly (CC-BY for data, permissive or copyleft choice for code);
robotics data with humans in frame needs consent/ethics notes.
- Long-term hosting: university archives, Zenodo-style DOI services, or IEEE
DataPort outlive lab NAS boxes and personal cloud links.
Review-time vs acceptance-time states
Under the double-anonymous policy (2026 cycle onward), the artifact has two
lives:
- Review state: anonymized hosting, no org/usernames in URLs, git history
squashed (history leaks author emails), license file present but
copyright-holder line deferred ("held for anonymity").
- Public state (post-acceptance): real hosting, authors restored, DOI
minted, README linking the IEEE Xplore entry, camera-ready pointing at the
permanent URL (coordinate with
icra-camera-ready).
Prepare both from the start; converting a name-riddled repo to anonymous form
in deadline week always misses something.
Safety as an artifact property
Robotics artifacts can move mass. Before anyone external runs your controller:
- Ship conservative default limits (velocity, torque, workspace bounds) and
make the paper's aggressive settings an explicit opt-in flag.
- Document the E-stop assumption and any human-proximity constraints in the
hardware protocol, not just the lab's tribal knowledge.
- State firmware/driver versions known to behave; a controller tuned on one
firmware can oscillate on another, and that failure lands on your artifact's
reputation.
Withholding honestly
Legitimate reasons to withhold layers exist (industrial partners, export
controls, safety of a hazardous procedure). The rule is disclosure: state in
the paper what is withheld and why, and maximize the released remainder — e.g.,
withheld weights but released training code and evaluation harness. Undisclosed
gaps discovered later cost more reputation than declared ones.
Packaging sequence
- Inventory the six layers; mark release state and blockers for each.
- Build the Docker/pinned environment; verify
make figure3 on a clean machine.
- Write
CLAIMS.md; separate rerunnable from hardware-audit claims.
- Add log-replay mode and sim twin for hardware-bound layers.
- Produce the anonymized review state; leak-check URLs, history, metadata.
- Stage the public state for acceptance day.
Output format
[Layer inventory] released: <layers> | partial: <layers> | withheld: <layers+reason>
[Five-minute test] clean-machine run of headline command: pass / fail
[CLAIMS.md] complete: y/n — hardware-only claims flagged: <list>
[Replay/sim twin] present: y/n
[Anonymity state] review-safe: y/n — leaks: <list>
[Hosting] review URL type + acceptance-day plan
