Hri Artifact Evaluation

HRI Artifact Evaluation

HRI's artifact and reproducibility culture is less formalized than software-engineering venues': there is not a long-standing, uniformly enforced ACM artifact-badge track at every HRI edition, and whether a given year runs a formal reproducibility/artifact evaluation is 待核实 per cycle. So treat artifact work at HRI as a credibility and openness move first, and follow a formal badge process only if this edition offers one. Either way, the artifact of an HRI paper is unusual: it is the study, not a runnable system that reproduces a number. This skill packages it honestly. It builds on hri-reproducibility and the shared kit in ../../resources/code/README.md.

First: does this edition run a formal evaluation?

• Check the current call for a reproducibility/artifact track, its badges, and its own deadline (待核实). If it exists, follow its instructions exactly.
• If ACM badges are offered, they follow the ACM scheme (Artifacts Available / Evaluated - Functional / Reusable / Results Reproduced). Map your package to whichever the edition supports.
• If no formal track runs, still release an open, well-documented artifact — reviewers and readers reward it, and it is the community norm even without a badge.

What an HRI artifact actually contains

Because the core evidence is a human study of an embodied robot, the artifact is rarely a one-command reproduction. Assemble:

• Study materials — questionnaires (items + scoring), interview guides, task scripts, consent-form summary, and coding schemes.
• Robot-behavior specification — the conditions/behaviors the robot exhibited: for autonomous behavior, the software version and parameters; for Wizard-of-Oz, the wizard's action space, protocol, and error rates (see hri-experiments).
• De-identified data — participant-level data with identifiers removed, plus a codebook.
• Analysis code — scripts that regenerate every reported statistic, figure, and table from the shared data, with pinned dependencies and fixed seeds.
• Pre-registration pointer and any deviation notes.

Make it reusable, within honest limits

• Reproducible = re-analyzable + re-runnable-as-a-method. An evaluator should be able to regenerate your results from the shared data and to understand the procedure well enough to run the study again — not to recreate your exact human responses. State this scope plainly.
• Document the environment. For analysis code, a README with dependency versions and a single entry point; a container helps if the toolchain is heavy.
• License it (an OSI-approved code license; a documented data-use/consent basis for the data).
• Archive with a DOI (OSF, Zenodo, figshare, or Software Heritage) rather than a personal homepage, so the link is permanent and citable.

Human-subjects constraints on the artifact

An HRI artifact contains people, which limits what can be shared:

• Share only data/media the participants consented to share for that use; if consent for public data or video was not obtained, share what you can and explain the gap honestly.
• De-identify thoroughly — free-text responses and video are the usual re-identification risks.
• Vulnerable-population studies (children, older adults, clinical) may permit little sharing; say so rather than force an unethical release.
• A partial, honest artifact with a clear rationale beats an over-shared one that breaches consent.

Camera-ready badge mechanics (if applicable)

• Badges, when a paper earns them, are displayed on the published version — coordinate with hri-camera-ready so any badge and the archive DOI appear correctly.
• Keep the anonymized review-time archive and the de-anonymized public archive separate; the public one carries author names, the review one must not.

Anti-patterns

• Overclaiming a "reproducible" human study as if it re-runs like a benchmark.
• "Data available on request" — treated as unavailable.
• Unlogged Wizard-of-Oz — the stimulus cannot be reproduced.
• Identifiable data or video released without consent for that use.
• Chasing a badge that does not exist this cycle instead of simply releasing a good artifact.

Output format

[Formal track?] does this edition run artifact/reproducibility evaluation + badges? (verify)
[Artifact contents] materials · robot-behavior spec (incl. WoZ) · de-identified data + codebook · analysis code
[Reproducibility scope] re-analysis + method replication; human responses framed as not re-runnable
[Openness] DOI archive · license · honest statement of consent-limited gaps
[Human-subjects] de-identified · shared only what consent allows
[Badge readiness] (if offered) which ACM badge(s) targeted · public vs review archive separated
[Fix queue] <ordered>

Source: brycewang-stanford/Awesome-Journal-Skills → HRI-Skills/skills/hri-artifact-evaluation/SKILL.md