CHI Artifact Evaluation
CHI's Papers track posted no separate artifact-evaluation committee or badge track
for the cycles checked (2026, 2027; confirm each year — 待核实 as a standing item).
That absence cuts both ways: nobody certifies your artifacts, and nobody but you
ensures a skeptical reviewer can inspect them. At CHI the "artifact" is rarely just
code — it is the prototype, the study instruments, the codebook, the dataset, and
the video evidence of the system working. Package for two audiences with opposite
needs: the anonymous reviewer with five minutes, and the future researcher with a
real use for your materials.
The CHI artifact inventory
| Artifact |
Reviewer's question |
Archival value after acceptance |
| Prototype / system code |
"Does the claimed interaction actually exist?" |
Others extend or compare against it |
| Video figure of real use |
"Does it work outside the authors' hands?" |
Permanent DL evidence (chi-supplementary) |
| Study instruments (guides, questionnaires, tasks) |
"Was the study what the paper says?" |
Direct reuse in replications |
| Codebook / analysis audit trail |
"Do the themes live in the data?" |
Methods teaching material |
| Dataset (de-identified) |
"Do the numbers re-derive?" |
Secondary analysis |
| Design files (3D prints, schematics, figma) |
"Could this be rebuilt?" |
Fabrication replication |
| Prompts / model configs for AI conditions |
"What system did participants face?" |
The only record once the API moves |
Inventory first, then decide per artifact: reviewed now, released later, or honestly
withheld with a reason (chi-reproducibility has the data-sharing ladder).
Reviewer-facing packaging (anonymous, five minutes)
The review-phase archive rides the single September deadline with everything else.
Design it so the first five minutes land:
- Root README with a claims map: "Claim in §5.1 →
analysis/h1_test.R →
Table 2." Three such lines are worth thirty pages of appendix.
- One command per claim where code is involved; vendor the environment
(
requirements.txt, renv.lock) because a reviewer will not debug pip.
- A demo path that does not require your hardware. If the contribution is a
physical device, the video figure is the reviewable artifact; the archive adds
schematics and firmware so the claim is auditable in principle.
- Anonymity end to end: no
.git, no metadata, no named accounts, anonymized
platform views only, usernames scrubbed from notebook outputs and file paths.
# Cold-simulate the reviewer on the exact ZIP you will upload
rm -rf /tmp/ae && unzip -q supplement.zip -d /tmp/ae && cd /tmp/ae
cat README* | head -30 # does the claims map appear immediately?
grep -rEil 'author|university|(^|[^a-z])lab' --include='*.md' . | head
time bash run_minimal_demo.sh # the five-minute budget is literal
Post-acceptance release is a separate product
At the publication-ready stage (February for CHI 2027), rebuild the artifact set
under real names for permanence, not for review:
- Deposit in a persistent home — the DL supplemental record, OSF, Zenodo, an
institutional archive — with a DOI; a lab URL is a dead link in five years.
- License deliberately: code (MIT/Apache/GPL), data and instruments (CC BY or
CC BY-NC), and record third-party constraints (stimuli copyrights, model terms
of service for cached AI outputs).
- Re-check de-identification harder than at review: public release is forever,
and participant re-identification harms real people. Where consent was narrow,
release the instruments and codebook instead of the data — respected practice.
- Tag the released version to match the camera-ready ("as-published"), then develop
onward in a separate branch; future readers need the paper's version, not HEAD.
Honest holes beat cosmetic completeness
A packaging note that says "the deployment used partner infrastructure we cannot
ship; this archive contains the full client, the API contract, and a mock server
reproducing the study conditions" earns more trust than a repo padded with dead
code that hides the same gap. Reviewers at CHI read many partial artifacts; what
they punish is discovering the gap themselves after the README implied completeness.
Timing inside the CHI year
- Build the claims-map README while writing §4–5 of the paper — it doubles as your
own claim-evidence audit (
chi-experiments).
- Freeze the review archive at T−1 week; sweep it with
chi-submission's checks.
- Diary the February release rebuild at acceptance time; rushed public releases in
the TAPS window are where consent violations happen.
Output format
[Inventory] <artifact: reviewed / released-later / withheld+reason, per row>
[Claims map] present in README: yes/no · claims covered: <n>/<n>
[Five-minute test] cold demo ran in <time> / failed at <step>
[Anonymity] archive clean: yes/no — <channels checked>
[Release plan] home: <DL/OSF/Zenodo> · license: <code/data> · consent re-check owner: <name>
1---2name: chi-artifact-evaluation3description: Use when packaging the artifacts behind an ACM CHI paper — prototypes, study instruments, codebooks, datasets, analysis code — for anonymous review scrutiny and for post-acceptance archival release, in a venue with no formal artifact-evaluation committee doing it for you.4---56# CHI Artifact Evaluation78CHI's Papers track posted no separate artifact-evaluation committee or badge track9for the cycles checked (2026, 2027; confirm each year — 待核实 as a standing item).10That absence cuts both ways: nobody certifies your artifacts, and nobody but you11ensures a skeptical reviewer can inspect them. At CHI the "artifact" is rarely just12code — it is the prototype, the study instruments, the codebook, the dataset, and13the video evidence of the system working. Package for two audiences with opposite14needs: the anonymous reviewer with five minutes, and the future researcher with a15real use for your materials.1617## The CHI artifact inventory1819| Artifact | Reviewer's question | Archival value after acceptance |20|---|---|---|21| Prototype / system code | "Does the claimed interaction actually exist?" | Others extend or compare against it |22| Video figure of real use | "Does it work outside the authors' hands?" | Permanent DL evidence (`chi-supplementary`) |23| Study instruments (guides, questionnaires, tasks) | "Was the study what the paper says?" | Direct reuse in replications |24| Codebook / analysis audit trail | "Do the themes live in the data?" | Methods teaching material |25| Dataset (de-identified) | "Do the numbers re-derive?" | Secondary analysis |26| Design files (3D prints, schematics, figma) | "Could this be rebuilt?" | Fabrication replication |27| Prompts / model configs for AI conditions | "What system did participants face?" | The only record once the API moves |2829Inventory first, then decide per artifact: reviewed now, released later, or honestly30withheld with a reason (`chi-reproducibility` has the data-sharing ladder).3132## Reviewer-facing packaging (anonymous, five minutes)3334The review-phase archive rides the single September deadline with everything else.35Design it so the first five minutes land:36371. **Root README** with a claims map: "Claim in §5.1 → `analysis/h1_test.R` →38 Table 2." Three such lines are worth thirty pages of appendix.392. **One command per claim** where code is involved; vendor the environment40 (`requirements.txt`, `renv.lock`) because a reviewer will not debug pip.413. **A demo path that does not require your hardware.** If the contribution is a42 physical device, the video figure *is* the reviewable artifact; the archive adds43 schematics and firmware so the claim is auditable in principle.444. **Anonymity end to end**: no `.git`, no metadata, no named accounts, anonymized45 platform views only, usernames scrubbed from notebook outputs and file paths.4647```bash48# Cold-simulate the reviewer on the exact ZIP you will upload49rm -rf /tmp/ae && unzip -q supplement.zip -d /tmp/ae && cd /tmp/ae50cat README* | head -30 # does the claims map appear immediately?51grep -rEil 'author|university|(^|[^a-z])lab' --include='*.md' . | head52time bash run_minimal_demo.sh # the five-minute budget is literal53```5455## Post-acceptance release is a separate product5657At the publication-ready stage (February for CHI 2027), rebuild the artifact set58under real names for permanence, not for review:5960- Deposit in a persistent home — the DL supplemental record, OSF, Zenodo, an61 institutional archive — with a DOI; a lab URL is a dead link in five years.62- License deliberately: code (MIT/Apache/GPL), data and instruments (CC BY or63 CC BY-NC), and record third-party constraints (stimuli copyrights, model terms64 of service for cached AI outputs).65- Re-check de-identification *harder* than at review: public release is forever,66 and participant re-identification harms real people. Where consent was narrow,67 release the instruments and codebook instead of the data — respected practice.68- Tag the released version to match the camera-ready ("as-published"), then develop69 onward in a separate branch; future readers need the paper's version, not HEAD.7071## Honest holes beat cosmetic completeness7273A packaging note that says "the deployment used partner infrastructure we cannot74ship; this archive contains the full client, the API contract, and a mock server75reproducing the study conditions" earns more trust than a repo padded with dead76code that hides the same gap. Reviewers at CHI read many partial artifacts; what77they punish is discovering the gap themselves after the README implied completeness.7879## Timing inside the CHI year8081- Build the claims-map README while writing §4–5 of the paper — it doubles as your82 own claim-evidence audit (`chi-experiments`).83- Freeze the review archive at T−1 week; sweep it with `chi-submission`'s checks.84- Diary the February release rebuild at acceptance time; rushed public releases in85 the TAPS window are where consent violations happen.8687## Output format8889```text90[Inventory] <artifact: reviewed / released-later / withheld+reason, per row>91[Claims map] present in README: yes/no · claims covered: <n>/<n>92[Five-minute test] cold demo ran in <time> / failed at <step>93[Anonymity] archive clean: yes/no — <channels checked>94[Release plan] home: <DL/OSF/Zenodo> · license: <code/data> · consent re-check owner: <name>95```
brycewang-stanford