brycewang-stanford PerCom Artifact Evaluation
Use this for reproducibility packaging. First, a cycle caveat: unlike SIGSOFT venues, PerCom has not historically run a mandatory formal artifact-evaluation track with a fixed badge set, and whether a given edition offers a reproducibility/badging track (e.g., IEEE Open Research Objects / Results Reproduced) is 待核实 — confirm on the current call. Regardless of whether a badge is offered, a well-packaged, de-identified sensing dataset and reproducible pipeline is a scored strength in the double-blind review and a lasting community contribution.
What "reproducible" means for a ubicomp sensing paper
Two deliverables, kept distinct:
- The anonymized review package (at submission): dataset link and code scrubbed of owner, testbed, and lab identity, for the double-blind reviewers.
- The public deposit (after acceptance): a de-identified dataset and code in a DOI-issuing archive under an open license — the version others cite and reuse, and the version any badge program evaluates.
What a ubicomp evaluator opens first
| Claim type | First thing inspected | Common failure caught |
|---|---|---|
| An activity/context recognizer | The script that regenerates cross-subject (LOSO) results | Only a pooled-accuracy script; no leave-one-subject-out path |
| A sensing dataset | The data itself + a datasheet (subjects, sensors, labels) | Link present, data missing; no de-identification described |
| A deployed system | A demo on bundled sample data | Only-runs-on-authors'-testbed; hardware not documented |
| A model result | Trained weights + inference on sample input | Requires the full raw dataset or private compute to run |
Assume an evaluator gives your package a bounded time budget on a clean machine with no access to your sensors or subjects. Design for the first ten minutes — a demo on bundled, de-identified sample data — to succeed.
Packaging plan
[Container] ship a Dockerfile or a pinned environment (requirements/lockfile); avoid
"install these 40 things by hand"
[README] one-screen orientation: what it is, install, run the demo, reproduce each claim,
expected runtime and outputs
[Datasheet] a dataset datasheet: subjects (count, relevant demographics), sensors (device,
firmware, sampling rate, placement), labels + protocol, and known biases
[Mapping] an explicit table: paper claim -> script -> expected result (with the LOSO split)
[Data] the de-identified dataset itself (or documented restricted-access), not just a query
[Ethics] IRB/consent status and the de-identification performed before release
[License] an open, DOI-issuing deposit (IEEE DataPort, Zenodo) so others can reuse and cite
De-identification is the ubicomp-specific bar
Human-subjects sensing data leaks identity in ways code does not: raw audio/video, GPS traces, timestamps that pinpoint a home, and even accelerometer gait can re-identify. Before any release:
- Remove or transform direct identifiers and re-identifying traces; document exactly what you did.
- Confirm your consent and IRB approval permit public release of the de-identified data — some approvals do not, and then the honest move is restricted access with a documented request path.
- Never ship a dataset that a subject did not consent to have published.
Worked vignette: a wearable-HAR dataset + recognizer
To make a HAR paper reproducible: ship a Docker image with the recognizer pre-built; a
run_demo.sh that classifies on a small bundled, de-identified sample in under a minute; a
reproduce/ directory whose scripts regenerate the leave-one-subject-out F1 table (not just a
pooled number) from logged features; a datasheet listing subjects, sensor placement, and sampling
rate; the de-identified dataset with a documented consent/IRB basis; and an open license with a
DOI. State honestly which results are turnkey and which need the full (slow) training run.
Calibration
- Whether a badge/reproducibility track runs, and which badges, is 待核实 per cycle — confirm on the current call rather than assuming a SIGSOFT-style scheme.
- The DOI-issuing deposit and datasheet are portable value even when no badge is offered.
- Anonymize the review package; de-identify (and confirm consent for) the public dataset — these are different obligations.
Output format
[Track status] formal reproducibility/badge track this cycle? yes/no/待核实
[Artifact role] anonymized review package / public de-identified deposit
[Contents] <recognizer/dataset/datasheet/scripts/ethics/license>
[Ten-minute test] does install + demo on bundled sample data succeed on a clean machine? yes/no
[Cross-subject reproduction] does a script regenerate the LOSO result? yes/no
[De-identification] documented + consent/IRB permits release? yes/no
[Fixes before deposit] <ordered list>