brycewang-stanford ICRA Experiments
Experimental evidence is where ICRA reviews are won and lost. The community's core suspicion is the cherry-picked demo: one lucky run, filmed once, presented as capability. The experiments section exists to prove the demo was not luck.
The evidence ladder
Climb as high as the claim requires; state the rung explicitly in the paper.
| Rung | Evidence | Supports claims like |
|---|---|---|
| 1 | Simulation only, single environment | "the formulation is feasible" |
| 2 | Simulation, randomized dynamics/scenes | "the method is robust in sim" |
| 3 | Real robot, controlled lab task | "works on hardware" |
| 4 | Real robot, varied objects/terrains/subjects | "generalizes physically" |
| 5 | Extended/field deployment, uncontrolled conditions | "works in the world" |
Simulation-only papers are not banned at ICRA, but a rung-2 evidence base cannot carry rung-4 language. The mismatch between claim altitude and evidence rung is the single most cited weakness in robotics reviews. If hardware is unreachable, scope the claims to simulation and say why the sim is trustworthy (validated dynamics, established benchmark, physics-accurate contact).
Trials, resets, and what a success rate means
- Pre-register the protocol privately: number of trials, success criterion, and abort rule before collecting headline numbers; report the criterion verbatim in the paper ("success = object lifted 10 cm and held 5 s").
- Trial counts: 5 trials per condition is anecdote, 10-25 is typical ICRA practice, more when variance is high. Report n everywhere, including captions.
- Resets are part of the method. If a human re-tees the object between trials, say so; automated-reset claims are different capability claims.
- No silent trial deletion. "Excluding two trials due to hardware fault" is fine when disclosed with cause; undisclosed exclusion is fabrication.
- Report dispersion, not just means: min/max or standard deviation over trials, and per-condition breakdowns rather than one pooled percentage.
Baseline fairness on hardware
Hardware baselines are expensive, so reviewers scrutinize the shortcuts:
- Run baselines on the same platform, same task instances, same sensor stream — a baseline run in simulation against a method run on hardware is not a comparison, it is two anecdotes.
- Tune the baseline with comparable effort and say what that effort was; "the baseline used default parameters" invites the fatal review sentence "the baseline was not tuned."
- Include the naive-but-strong baseline (a well-engineered classical planner or PID stack). Beating only learned rivals while a 1990s controller would have matched the result is a known embarrassment pattern at robotics venues.
- When a competitor's code is closed, reimplement and mark results "reimplementation," or compare on published numbers with the setup delta stated.
Sim-to-real claims
If the paper trains in simulation and deploys on hardware, quantify the gap, never just bridge it rhetorically:
- Report the same metric in sim and real side by side; the delta is a result.
- Name the randomization/adaptation mechanism and ablate it — "domain randomization" as an unexamined incantation draws fire.
- Zero-shot vs fine-tuned transfer must be labeled; ten minutes of on-robot fine-tuning is legitimate but is a different claim.
Failure analysis as a first-class section
ICRA reviewers trust papers that dissect their failures more than papers that report none. Budget for a failure table:
Failure taxonomy (example structure)
F1 perception: slip not detected (4/50 trials) — cause: torque noise
under 0.3 N·m threshold; mitigation: none in current sensing
F2 planning: recovery pose in collision (2/50) — cause: stale octomap
F3 hardware: gripper timeout (1/50) — excluded (disclosed), watchdog fixed
Reported: 43/50 success (86%); all 7 failures categorized above.
A reviewer reading this knows the team understands its own system. The video
attachment should show at least one failure case for the same reason
(see icra-supplementary).
Ablations with hardware budgets
Full ablation grids are a simulation luxury. The hardware-honest pattern:
- Run the complete grid in simulation; run on hardware only the two or three ablations that test the paper's central mechanism.
- Say exactly this in the paper — reviewers accept the economics when the choice of which ablations earned hardware time is justified.
- Never present a simulation ablation in a way that lets a reader mistake it for a hardware result; label every table row's regime.
Statistics that fit robotics sample sizes
- With n = 10-25 trials per condition, prefer exact binomial confidence intervals for success rates over normal approximations.
- Paired designs (same object set for method and baseline) plus a paired test beat unpaired comparisons at the same n.
- Do not run twelve significance tests on six pages; pick the one comparison the claim rests on and treat the rest descriptively.
- Learning components: report seeds and per-seed outcomes for training, then hold the deployed policy fixed across hardware trials so trial variance is physical, not training noise.
Pre-deadline experiment audit
- Claim altitude vs evidence rung matched for every abstract claim?
- Success criterion stated verbatim; n visible in every table/caption?
- Baselines: same hardware, tuned, includes strong classical option?
- Sim-vs-real deltas reported numerically where transfer is claimed?
- Failure taxonomy present, counts summing to total trials?
- Statistics: intervals suited to small n; one load-bearing test?
Output format
[Evidence rung] <1-5> vs claim altitude <1-5> — mismatch: <y/n>
[Trial audit] n per condition, criterion stated, exclusions disclosed
[Baseline audit] hardware-matched? tuned? classical baseline present?
[Transfer audit] sim/real deltas numeric? adaptation labeled?
[Failure analysis] taxonomy present? counts reconcile?
[Single weakest table/figure] <which and why>