brycewang-stanford RSS Experiments
Design experiments that test the paper's claim, not experiments that showcase the system. At RSS the evaluation section is where the scientific claim either becomes falsifiable or is exposed as marketing.
Start from the claim, derive the conditions
Write the claim, then derive what evidence its logical form demands:
| Claim form | Evidence the form demands |
|---|---|
| "X causes the improvement" | Ablation removing only X; everything else frozen |
| "the gain transfers" | Held-out tasks/objects/platforms named before running |
| "method M has property P" | The condition where P would break, deliberately tested |
| "T is the bottleneck" | Failure attribution showing T dominates the failure mass |
| "faster/cheaper at equal quality" | Matched-quality comparison, not best-vs-default |
An experiment matrix that cannot be traced back to a claim row is footage, not evidence — cut it or move it to the supplement.
Hardware trial protocol
- Pre-register internally: success criterion, reset procedure, object/task distribution, and stopping rule written down before the campaign. Post-hoc criteria are how demo bias enters honest labs.
- Per-condition counts in every table. "n = 25 per object, 6 objects" is auditable; "extensive trials" is not.
- Report the denominator. All attempts count — aborted runs, resets, operator interventions. A human silently rescuing the robot between trials is part of the system and must appear in the protocol.
- Small-n honesty: hardware budgets cap trials, so use interval estimates suited to small samples and let the language match their width. Twenty trials support "in our setting"; they do not support "reliably in general."
Simulation and hardware: split the ledger
- Label every number sim or real; never average across the boundary.
- Simulation earns its place by scale (sweeps, distributions) or danger (failure regimes hardware cannot safely visit); hardware earns the claim's headline.
- If the claim mentions the physical world, at least one decisive result must be physical. A sim-only paper must scope its claim to simulation explicitly.
- When sim and real disagree, that gap is a finding — report it, do not tune it away silently.
Failure attribution, the RSS signature move
A distribution over failure causes is often more scientifically valuable than the success rate above it. Build one:
- Define 4-6 mutually exclusive failure categories before the campaign (perception, planning, timing, slip, hardware fault, other).
- Label every failed trial at collection time, from logs — retrospective labeling from memory drifts optimistic.
- Report the distribution per condition; a method that shifts failure mass between categories is telling you how it works.
Trial-budget arithmetic (do this before the campaign)
Hardware time is the binding constraint, so budget it explicitly:
conditions: 2 methods x 3 environments x (1 ablation + 1 transfer) = 10
trials/condition: 25 -> 250 attempts
minutes/attempt (incl. reset): 6 -> 25 robot-hours
overhead (faults, recalibration, re-runs): x1.5 -> ~38 robot-hours
robot access: 4 h/day -> ~10 working days, before any surprise
If the arithmetic says the matrix cannot finish before the January freeze, shrink the matrix, not the per-condition counts — fewer conditions with auditable n beat a full grid of anecdotes. Reserve the final week for zero data collection.
Reporting floor
Non-negotiables for the results section, independent of subfield:
- Trials per condition and total attempts, including discards, with discard rules.
- Success criterion stated operationally (what sensor reading / judge decides).
- Dispersion for every stochastic number — interval, quartiles, or all raw points when n is small; never a bare mean.
- Failure distribution table matched to the labeled categories.
- Wall-clock and compute for learned components; trial duration for hardware.
- Which numbers are sim and which are real, in the table itself, not the caption.
Baseline fairness
- Give baselines the same tuning budget, sensor stream, and reset quality as the proposed method, and say so in one sentence.
- Prefer the strongest published configuration over a reimplementation you cannot make fast; if reimplementing, report your version's score on a setting where the original published numbers exist.
Vignette: turning a demo matrix into a claim test
A draft evaluates a new whole-body controller on eight household tasks and reports success rates — a demo matrix. The claim, once written down, is "torque-limit awareness, not trajectory optimality, drives the reliability gain." The redesign: (1) an ablation running the same controller with torque-awareness disabled, same eight tasks; (2) a stress condition where torque limits bind hard (heavy objects) versus one where they never bind — the claim predicts the gain concentrates in the first; (3) failure attribution distinguishing torque-saturation failures from tracking failures. Three conditions replaced five decorative tasks, and the paper gained a falsifiable spine without new hardware.
Output format
[Claim -> condition map] <each claim row -> experiment>
[Protocol status] pre-registered / drifting / post-hoc
[Counts] <per-condition n, denominators, interventions>
[Sim/real ledger] clean split / mixed (fix)
[Failure attribution] <categories + dominant mass>
[Decisive missing run] <the one experiment to add next>
Source: brycewang-stanford/Awesome-Journal-Skills → RSS-Skills/skills/rss-experiments/SKILL.md