ICRA Reproducibility
Reproducibility at a robotics conference means something different than at an ML
venue: no reader can rerun your hardware trials without your robot, your room,
and your objects. The achievable standard is therefore two-tier — rerunnable
(code, simulation, analysis) and auditable (hardware evidence logged well
enough that a skeptic can verify you did what you claim). ICRA papers earn trust
by being explicit about which tier each result sits in.
The specification ledger
A robotics result is unreproducible if the platform is underspecified. The paper
(body, since there is no appendix escape) plus released materials should pin:
| Layer |
Must specify |
Example of "enough" |
| Robot |
model, DOF, payload, firmware version |
"UR5e, PolyScope 5.11" |
| End effector |
model, mods, wear state if relevant |
"Robotiq 2F-85, stock pads" |
| Sensors |
model, resolution, rate, mounting frame |
"D435i @ 640×480, 30 Hz, wrist" |
| Compute |
on-board vs off-board, GPU, latency path |
"off-board 3080, 5G-free wired" |
| Middleware |
ROS distro, key package versions |
"ROS 2 Humble, MoveIt 2.5.4" |
| Control |
loop rates, gains or their source |
"impedance @ 500 Hz, gains in repo" |
| Calibration |
method + residual error |
"hand-eye via [ref], 1.8 mm RMS" |
| Objects/scenes |
identity, source, dimensions |
"YCB subset listed in Tab. II" |
Standard object sets (YCB and similar) exist precisely so that "graspable
household objects" can be a checkable claim; use them or publish the object
specs.
Logging: the rosbag is the lab notebook
- Record every hardware session — including failed ones — with topic-complete
rosbags (or vendor-equivalent logs) plus an untouched camera view.
- One log per trial, named by protocol:
trial_<cond>_<idx>_<date>.bag; the
analysis pipeline should consume these logs, so paper numbers regenerate from
raw data with one command.
- Keep a session sheet: date, operator, firmware state, anomalies. Two months
later, this distinguishes "the gripper was recalibrated between Tables II and
III" from unexplainable drift.
- The video attachment should be cut from these recorded sessions, which
guarantees footage corresponds to logged, reported trials.
Determinism where it exists, honesty where it doesn't
Hardware trials are not seed-reproducible — contact, cables, lighting, and
thermal drift see to that. Split the claims accordingly:
- Deterministic tier: training runs, simulation experiments, and analysis
scripts get pinned seeds, pinned dependency versions, and containerized
environments. State seeds and repetition counts.
- Stochastic tier: hardware trials get protocol reproducibility — the
documented procedure another lab could follow — plus dispersion reporting.
- Never write "results are fully reproducible" over a hardware table; write
"code and simulation reproduce Tables I-II; hardware protocol and logs for
Tables III-IV are released for audit."
Release scaffold
repo/
├── README.md # claim → command map (Table III = run_hw_analysis.sh)
├── env/ # Dockerfile or rosdep/lockfiles, pinned
├── sim/ # simulation experiments, seeds fixed
├── control/ # controllers/planners as ROS packages
├── hw_protocol.md # step-by-step trial procedure, success criterion,
│ # reset procedure, safety notes
├── logs/ # rosbags or download script (size!), session sheets
├── analysis/ # regenerates every figure/table from logs/
└── video/ # raw clips ↔ trial IDs mapping
Anonymize the repository for review under the double-anonymous policy (2026
cycle onward): anonymized hosting or an artifact stripped of names — an org URL
is an identity leak (see icra-submission).
Compute and latency reporting
Robotics adds a real-time dimension absent from ML reproducibility norms:
- Report where each computation ran (on-board CPU/embedded GPU vs off-board
workstation) and the link between them; an off-board 3090 over gigabit wire
is a materially different system than on-board inference.
- State end-to-end latency per pipeline stage (sensing → perception → decision
→ actuation) at least once; "runs in real time" is unverifiable without it.
- Training compute (GPU-hours, hardware) belongs in the paper for learned
components — cheap to report, increasingly expected by review forms.
Availability statement patterns
Honest, specific statements outperform boilerplate promises:
- "Code, simulation environments, and trial logs: (anonymized for
review; public on acceptance)."
- "The gripper design files are released; the arm is commercial (UR5e)."
- "Raw rosbags total 1.2 TB; per-trial extracted features and a 40 GB sample
are hosted, full logs on request."
- Avoid: "code will be released upon publication" with nothing at review time —
reviewers discount unverifiable promises, and some cycles ask availability
questions directly on the review form (verify the current form).
Pre-submission reproducibility pass
- Specification ledger complete in-paper (table above, compressed)?
- Every reported number regenerates from logs via one documented command?
- Seeds/versions pinned for the deterministic tier; container builds clean?
- Hardware protocol written so a stranger could run trial 1 tomorrow?
- Release scaffold anonymized; links leak-checked?
- Availability statement matches what actually exists today?
Output format
[Tier split] rerunnable: <tables/figs> | auditable: <tables/figs>
[Spec ledger] missing rows: <list or none>
[Log coverage] trials logged <n>/<n>, analysis regenerates: y/n
[Determinism] seeds/versions pinned for sim+training: y/n
[Release] scaffold complete? anonymized? size plan?
[Statement] honest-specific / boilerplate-risk — rewrite: <line>
1---2name: icra-reproducibility3description: Use when hardening an ICRA paper's reproducibility — specifying robot platform, firmware, ROS and driver versions, control rates, and sensor calibration; logging rosbags and seeds; separating what others can rerun (code, sim) from what they can only audit (your hardware trials); and writing honest availability statements.4metadata:5 source: https://skillmd.com/skills/brycewang-stanford/icra-reproducibility6---78# ICRA Reproducibility910Reproducibility at a robotics conference means something different than at an ML11venue: no reader can rerun your hardware trials without your robot, your room,12and your objects. The achievable standard is therefore two-tier — **rerunnable**13(code, simulation, analysis) and **auditable** (hardware evidence logged well14enough that a skeptic can verify you did what you claim). ICRA papers earn trust15by being explicit about which tier each result sits in.1617## The specification ledger1819A robotics result is unreproducible if the platform is underspecified. The paper20(body, since there is no appendix escape) plus released materials should pin:2122| Layer | Must specify | Example of "enough" |23|---|---|---|24| Robot | model, DOF, payload, firmware version | "UR5e, PolyScope 5.11" |25| End effector | model, mods, wear state if relevant | "Robotiq 2F-85, stock pads" |26| Sensors | model, resolution, rate, mounting frame | "D435i @ 640×480, 30 Hz, wrist" |27| Compute | on-board vs off-board, GPU, latency path | "off-board 3080, 5G-free wired" |28| Middleware | ROS distro, key package versions | "ROS 2 Humble, MoveIt 2.5.4" |29| Control | loop rates, gains or their source | "impedance @ 500 Hz, gains in repo" |30| Calibration | method + residual error | "hand-eye via [ref], 1.8 mm RMS" |31| Objects/scenes | identity, source, dimensions | "YCB subset listed in Tab. II" |3233Standard object sets (YCB and similar) exist precisely so that "graspable34household objects" can be a checkable claim; use them or publish the object35specs.3637## Logging: the rosbag is the lab notebook3839- Record every hardware session — including failed ones — with topic-complete40 rosbags (or vendor-equivalent logs) plus an untouched camera view.41- One log per trial, named by protocol: `trial_<cond>_<idx>_<date>.bag`; the42 analysis pipeline should consume these logs, so paper numbers regenerate from43 raw data with one command.44- Keep a session sheet: date, operator, firmware state, anomalies. Two months45 later, this distinguishes "the gripper was recalibrated between Tables II and46 III" from unexplainable drift.47- The video attachment should be cut from these recorded sessions, which48 guarantees footage corresponds to logged, reported trials.4950## Determinism where it exists, honesty where it doesn't5152Hardware trials are not seed-reproducible — contact, cables, lighting, and53thermal drift see to that. Split the claims accordingly:5455- **Deterministic tier:** training runs, simulation experiments, and analysis56 scripts get pinned seeds, pinned dependency versions, and containerized57 environments. State seeds and repetition counts.58- **Stochastic tier:** hardware trials get *protocol* reproducibility — the59 documented procedure another lab could follow — plus dispersion reporting.60- Never write "results are fully reproducible" over a hardware table; write61 "code and simulation reproduce Tables I-II; hardware protocol and logs for62 Tables III-IV are released for audit."6364## Release scaffold6566```text67repo/68├── README.md # claim → command map (Table III = run_hw_analysis.sh)69├── env/ # Dockerfile or rosdep/lockfiles, pinned70├── sim/ # simulation experiments, seeds fixed71├── control/ # controllers/planners as ROS packages72├── hw_protocol.md # step-by-step trial procedure, success criterion,73│ # reset procedure, safety notes74├── logs/ # rosbags or download script (size!), session sheets75├── analysis/ # regenerates every figure/table from logs/76└── video/ # raw clips ↔ trial IDs mapping77```7879Anonymize the repository for review under the double-anonymous policy (202680cycle onward): anonymized hosting or an artifact stripped of names — an org URL81is an identity leak (see `icra-submission`).8283## Compute and latency reporting8485Robotics adds a real-time dimension absent from ML reproducibility norms:8687- Report where each computation ran (on-board CPU/embedded GPU vs off-board88 workstation) and the link between them; an off-board 3090 over gigabit wire89 is a materially different system than on-board inference.90- State end-to-end latency per pipeline stage (sensing → perception → decision91 → actuation) at least once; "runs in real time" is unverifiable without it.92- Training compute (GPU-hours, hardware) belongs in the paper for learned93 components — cheap to report, increasingly expected by review forms.9495## Availability statement patterns9697Honest, specific statements outperform boilerplate promises:9899- "Code, simulation environments, and trial logs: <link> (anonymized for100 review; public on acceptance)."101- "The gripper design files are released; the arm is commercial (UR5e)."102- "Raw rosbags total 1.2 TB; per-trial extracted features and a 40 GB sample103 are hosted, full logs on request."104- Avoid: "code will be released upon publication" with nothing at review time —105 reviewers discount unverifiable promises, and some cycles ask availability106 questions directly on the review form (verify the current form).107108## Pre-submission reproducibility pass1091101. Specification ledger complete in-paper (table above, compressed)?1112. Every reported number regenerates from logs via one documented command?1123. Seeds/versions pinned for the deterministic tier; container builds clean?1134. Hardware protocol written so a stranger could run trial 1 tomorrow?1145. Release scaffold anonymized; links leak-checked?1156. Availability statement matches what actually exists today?116117## Output format118119```text120[Tier split] rerunnable: <tables/figs> | auditable: <tables/figs>121[Spec ledger] missing rows: <list or none>122[Log coverage] trials logged <n>/<n>, analysis regenerates: y/n123[Determinism] seeds/versions pinned for sim+training: y/n124[Release] scaffold complete? anonymized? size plan?125[Statement] honest-specific / boilerplate-risk — rewrite: <line>126```