Hri Reproducibility

HRI Reproducibility

Reproducibility means something specific — and harder — at HRI than at a benchmark-driven venue. You cannot "re-run" a study of an embodied robot and real people the way you re-run a model on a test set: the participants, the room, the exact robot, and the moment are gone. So HRI reproducibility is about making the procedure, stimulus, and analysis transparent enough that another team could replicate the method and readers can audit your inferences. This skill builds that story; it pairs with the shared kit in ../../resources/code/README.md.

What "reproducible" can and cannot mean here

Be honest about the limits, and design the artifact around what is reproducible:

• Reproducible (aim for this): the study protocol, the robot behavior/stimulus, the measures, the analysis code, and the de-identified data — enough to re-analyze your results and to run the same study elsewhere.
• Not reproducible (say so): the exact human responses. A replication is a new sample; framing it otherwise overclaims. State this plainly rather than let a reviewer catch the overclaim.

The replication package for an HRI study

Assemble, and reference from the paper:

• Study materials — consent-form summary, instructions/scripts, questionnaires with exact items and scoring, interview guides, and coding schemes for qualitative work.
• Robot-behavior specification — the behaviors/conditions the robot exhibited, at enough granularity to reproduce the stimulus: for autonomous behavior, the software version and parameters; for Wizard-of-Oz, the wizard's action space, protocol/script, and error rates (see hri-experiments).
• De-identified data — participant-level data with identifiers removed, plus a codebook. Scrub free-text and metadata that could re-identify participants.
• Analysis code — scripts that reproduce every reported statistic, figure, and table from the shared data, with fixed seeds and pinned dependencies.
• Pre-registration — a link/ID if you pre-registered the hypotheses and analysis, with any deviations documented.

Report the things that make a study re-runnable

• Sample and procedure in reproducible detail: recruitment, inclusion criteria, N and how it was determined, session flow, timing, compensation.
• Apparatus: the robot platform and version, sensors, environment, and anything that would change the interaction if swapped.
• Wizard-of-Oz disclosure and constraints, so the stimulus is not an unlogged human performance.
• Analysis decisions: exclusions and why, transformations, the exact tests, and the confirmatory/exploratory split.

De-identification and the video

HRI artifacts contain people, which raises obligations a code-only artifact does not:

• De-identify data before sharing; get consent for data sharing at study time — retrofitting it is usually impossible.
• The video figure and any recordings need consent for the use you are making (review-time sharing, public archival, or conference presentation are different permissions). For review, blur faces and mute identifying audio if consent for public use was not obtained (see hri-supplementary).
• Vulnerable populations (children, older adults, clinical) demand extra care in what is shared at all.

Openness at HRI (calibrated to the venue)

HRI's open-science and artifact-badge culture is less formalized than SIGSOFT's — there is not a long-standing, uniformly enforced ACM badge track for every edition (待核实 per cycle). So treat openness as a credibility and community-norm move, not a checkbox:

• Share what ethics allows via a DOI-issuing archive (e.g., OSF, Zenodo, figshare); avoid a personal homepage link that both rots and de-anonymizes.
• Where confidentiality or human-subjects constraints prevent full sharing, say what and why — an honest limitation reads far better than silence or "available on request."
• If the edition does run an artifact/reproducibility evaluation, follow it and see hri-artifact-evaluation.

Anonymization for double-blind

Everything above must be anonymized at review time: strip author/institution names from materials and code, host the archive behind an anonymizing view, and scrub the video and data of identity leaks. See hri-submission for the mechanical sweep.

Anti-patterns HRI reviewers flag

• Overclaiming reproducibility — implying a human study can be exactly re-run.
• Unlogged Wizard-of-Oz — no way to reproduce the stimulus.
• "Data available on request" — treated as not available.
• Identifiable data or video shared without consent for that use.
• Analysis not reproducible from the shared data — figures that no script regenerates.

Output format

[Repro scope] procedure/stimulus/analysis reproducible? human responses correctly framed as not?
[Package] materials · robot-behavior spec (incl. WoZ) · de-identified data + codebook · analysis code
[Pre-registration] link/ID · deviations documented?
[De-identification] data scrubbed · video/audio consent for the use made?
[Openness] DOI archive · honest statement of what cannot be shared and why
[Anonymization] materials/code/archive/video anonymized for review?
[Fix queue] <ordered>

Source: brycewang-stanford/Awesome-Journal-Skills → HRI-Skills/skills/hri-reproducibility/SKILL.md