CoRL Reproducibility
Robot-learning papers have a split reproducibility problem: the training half is
software and can in principle be rerun anywhere, while the hardware half is a
physical setup nobody else owns. Strong CoRL papers treat these halves
differently — the software half is made rerunnable, the hardware half is made
auditable — and say plainly which is which.
The rerunnable half: simulation and training
The silent reproducibility killers in this field are version-shaped:
- Simulator versions change physics. Contact solvers, default damping, and
collision margins shift between releases of MuJoCo, Isaac, PyBullet, and
friends; a policy's success rate is a function of the simulator build. Pin the
exact version and any physics-relevant flags.
- Environment wrappers drift. Task-suite repositories (benchmark forks,
custom reward shims) move under you; record the commit hash of every env repo,
not just your own.
- GPU nondeterminism. cuDNN autotuning and atomics make bit-identical
training runs unrealistic — so define reproducibility at the distribution
level: same configs + fresh seeds should land inside your reported seed spread.
# repro-manifest.yaml — ship at the repo root; one block per results table
table_3:
code_commit: "" # your repo @ exact hash
env_suite: {repo: "", commit: "", sim: "mujoco==X.Y.Z", physics_flags: []}
training: {config: "configs/table3.yaml", seeds: [0,1,2,3,4], gpu: "1xA100-80GB", hours: 14}
data: {demos: "dataset-v2 (1,204 episodes)", url_or_status: "", license: ""}
checkpoints: {released: true, path: "ckpts/table3/", selection_rule: "last epoch, no eval peeking"}
evaluation: {script: "eval/run.py", episodes_per_task: 50, init_state_list: "eval/states.json"}
expected_tolerance: "per-task success within ±4 pts of Table 3 mean"
The selection_rule line matters more than it looks: checkpoint selection via
test-set peeking is the field's quiet irreproducibility engine, and stating the
rule is the cheapest credibility purchase available.
The auditable half: hardware
Nobody will re-run your robot, so the goal is that a skeptical expert could
verify the experiment happened as described and rebuild an equivalent rig:
| What to document |
Why an auditor needs it |
| Robot model, end-effector, firmware/driver versions |
Behavior differs across firmware, not just robots |
| Sensor models, mounting poses, calibration procedure |
Camera extrinsics silently dominate visuomotor results |
| Control interface: frequency, action space, safety filters |
"30 Hz end-effector deltas" vs "torque control" are different papers |
| Deployed compute + inference latency |
Policy behavior is latency-dependent |
| Scene inventory: objects (make/size), fixtures, lighting |
Enables an equivalent-rig rebuild and honest comparison |
| Raw episode logs and unedited evaluation video |
The audit trail for every printed success rate |
Log every evaluation episode at capture time (timestamped video plus a CSV of
outcomes); retrofitting an audit trail after reviews ask for it is impossible.
Data and checkpoint release
- Demonstration datasets are results: release episode counts, collection method
(teleop rig, scripted, crowdsourced), operator count, and filtering rules.
- Released checkpoints let others reproduce evaluation even when training is
too expensive to repeat — for large policies this is often the highest-value
artifact you can ship.
- If data or weights cannot be released (proprietary platform, human-subject
footage), say so in the paper with the reason, and release what remains:
configs, eval scripts, sim environments, and metrics logs. A precise
partial-release statement outperforms a vague "code available upon request."
During review vs after acceptance
- During review everything must be anonymous (see
corl-submission): anonymized
repo mirrors, no lab-identifying video, no cloud buckets with named projects.
- The 2026 camera-ready pipeline has a sharp constraint: PMLR does not accept
video as supplementary material, so post-acceptance videos, code, and data
live on external hosting (project site, GitHub, archive), linked from the main
text (corl.org author instructions, read 2026-07-08). Plan the public homes of
artifacts before the October camera-ready deadline, and prefer DOI-stamped
archives for anything you cite as permanent.
- No formal reproducibility checklist was verified for the 2026 cycle in this
pack (待核实) — but reviewer expectations enforce one informally; use the
manifest above regardless of what the form requires.
Availability statement patterns
Strong: "Code, training configs, evaluation scripts, and the 1,204-episode
teleop dataset: <URL>. Checkpoints for Tables 2-4: <URL>. Hardware
evaluations are documented in Appendix C (rig spec, logs, uncut video);
the platform itself cannot be redistributed."
Weak: "Code will be released upon acceptance." (unverifiable promise)
Broken: "Results reproducible with standard settings." (no artifact at all)
Pre-submission audit
[ ] repro-manifest present; one block per headline table
[ ] Simulator/env/driver versions + commits pinned everywhere
[ ] Seed policy stated; checkpoint selection rule stated
[ ] Hardware rig spec complete enough for an equivalent rebuild
[ ] Episode-level logs + uncut eval video archived internally
[ ] Data/checkpoint release plan with named blockers, if any
[ ] Anonymous during review; external hosting plan ready for camera-ready
Verify the live cycle's supplementary and camera-ready rules at
https://www.corl.org/contributions/instruction-for-authors before promising any
artifact channel — PMLR-side constraints and CoRL-side forms both change yearly.
1---2name: corl-reproducibility3description: Use when making a CoRL robot-learning paper reproducible — pinning simulator and driver versions, releasing training configs, demonstration data and checkpoints, documenting hardware setups that cannot be rerun, seed policy, evaluation scripts, and honest availability statements for code, data, and robot platforms.4---56# CoRL Reproducibility78Robot-learning papers have a split reproducibility problem: the *training* half is9software and can in principle be rerun anywhere, while the *hardware* half is a10physical setup nobody else owns. Strong CoRL papers treat these halves11differently — the software half is made **rerunnable**, the hardware half is made12**auditable** — and say plainly which is which.1314## The rerunnable half: simulation and training1516The silent reproducibility killers in this field are version-shaped:1718- **Simulator versions change physics.** Contact solvers, default damping, and19 collision margins shift between releases of MuJoCo, Isaac, PyBullet, and20 friends; a policy's success rate is a function of the simulator build. Pin the21 exact version and any physics-relevant flags.22- **Environment wrappers drift.** Task-suite repositories (benchmark forks,23 custom reward shims) move under you; record the commit hash of every env repo,24 not just your own.25- **GPU nondeterminism.** cuDNN autotuning and atomics make bit-identical26 training runs unrealistic — so define reproducibility at the *distribution*27 level: same configs + fresh seeds should land inside your reported seed spread.2829```yaml30# repro-manifest.yaml — ship at the repo root; one block per results table31table_3:32 code_commit: "" # your repo @ exact hash33 env_suite: {repo: "", commit: "", sim: "mujoco==X.Y.Z", physics_flags: []}34 training: {config: "configs/table3.yaml", seeds: [0,1,2,3,4], gpu: "1xA100-80GB", hours: 14}35 data: {demos: "dataset-v2 (1,204 episodes)", url_or_status: "", license: ""}36 checkpoints: {released: true, path: "ckpts/table3/", selection_rule: "last epoch, no eval peeking"}37 evaluation: {script: "eval/run.py", episodes_per_task: 50, init_state_list: "eval/states.json"}38expected_tolerance: "per-task success within ±4 pts of Table 3 mean"39```4041The `selection_rule` line matters more than it looks: checkpoint selection via42test-set peeking is the field's quiet irreproducibility engine, and stating the43rule is the cheapest credibility purchase available.4445## The auditable half: hardware4647Nobody will re-run your robot, so the goal is that a skeptical expert could48*verify the experiment happened as described* and *rebuild an equivalent rig*:4950| What to document | Why an auditor needs it |51|---|---|52| Robot model, end-effector, firmware/driver versions | Behavior differs across firmware, not just robots |53| Sensor models, mounting poses, calibration procedure | Camera extrinsics silently dominate visuomotor results |54| Control interface: frequency, action space, safety filters | "30 Hz end-effector deltas" vs "torque control" are different papers |55| Deployed compute + inference latency | Policy behavior is latency-dependent |56| Scene inventory: objects (make/size), fixtures, lighting | Enables an equivalent-rig rebuild and honest comparison |57| Raw episode logs and unedited evaluation video | The audit trail for every printed success rate |5859Log every evaluation episode at capture time (timestamped video plus a CSV of60outcomes); retrofitting an audit trail after reviews ask for it is impossible.6162## Data and checkpoint release6364- Demonstration datasets are results: release episode counts, collection method65 (teleop rig, scripted, crowdsourced), operator count, and filtering rules.66- Released checkpoints let others reproduce *evaluation* even when training is67 too expensive to repeat — for large policies this is often the highest-value68 artifact you can ship.69- If data or weights cannot be released (proprietary platform, human-subject70 footage), say so in the paper with the reason, and release what remains:71 configs, eval scripts, sim environments, and metrics logs. A precise72 partial-release statement outperforms a vague "code available upon request."7374## During review vs after acceptance7576- During review everything must be anonymous (see `corl-submission`): anonymized77 repo mirrors, no lab-identifying video, no cloud buckets with named projects.78- The 2026 camera-ready pipeline has a sharp constraint: **PMLR does not accept79 video as supplementary material**, so post-acceptance videos, code, and data80 live on external hosting (project site, GitHub, archive), linked from the main81 text (corl.org author instructions, read 2026-07-08). Plan the public homes of82 artifacts before the October camera-ready deadline, and prefer DOI-stamped83 archives for anything you cite as permanent.84- No formal reproducibility checklist was verified for the 2026 cycle in this85 pack (待核实) — but reviewer expectations enforce one informally; use the86 manifest above regardless of what the form requires.8788## Availability statement patterns8990```text91Strong: "Code, training configs, evaluation scripts, and the 1,204-episode92 teleop dataset: <URL>. Checkpoints for Tables 2-4: <URL>. Hardware93 evaluations are documented in Appendix C (rig spec, logs, uncut video);94 the platform itself cannot be redistributed."95Weak: "Code will be released upon acceptance." (unverifiable promise)96Broken: "Results reproducible with standard settings." (no artifact at all)97```9899## Pre-submission audit100101```text102[ ] repro-manifest present; one block per headline table103[ ] Simulator/env/driver versions + commits pinned everywhere104[ ] Seed policy stated; checkpoint selection rule stated105[ ] Hardware rig spec complete enough for an equivalent rebuild106[ ] Episode-level logs + uncut eval video archived internally107[ ] Data/checkpoint release plan with named blockers, if any108[ ] Anonymous during review; external hosting plan ready for camera-ready109```110111Verify the live cycle's supplementary and camera-ready rules at112https://www.corl.org/contributions/instruction-for-authors before promising any113artifact channel — PMLR-side constraints and CoRL-side forms both change yearly.
brycewang-stanford