IPSN Artifact Evaluation
Use this for the artifact track. IPSN had a strong artifact culture — a Best Research Artifact
Award alongside the Best Paper Award — and its artifacts are unusual because they mix hardware,
firmware, and data, not just code. Two things to internalize: badges and the award are earned by
evaluators actually using your package, and the review artifact (anonymized, for the paper's
reviewers) is not the same deliverable as the public artifact (de-anonymized, DOI-archived). Because
IPSN merged into SenSys, confirm the current badge set, the award's persistence, and the deadline on
the successor call (待核实).
Badges and the award (verify the current set)
| Target |
What it certifies |
What earns it |
| Artifacts Available (ACM) |
The artifact is permanently, publicly retrievable |
Deposit in a DOI-issuing archive (Zenodo, IEEE DataPort, figshare, Software Heritage) |
| Artifacts Evaluated - Functional |
The artifact runs and does what the paper says |
A clean-machine (or emulated) install, a demo, documented expected outputs |
| Artifacts Evaluated - Reusable |
Others can build on it |
The Functional bar plus careful docs, structure, licensing, and board/firmware detail |
| Results Reproduced |
An evaluator reproduced key results |
A turnkey analysis path from raw traces to the headline numbers |
| IPSN Best Research Artifact Award |
Community recognition of an exceptional artifact |
A package an evaluator can genuinely run and reuse, hardware caveats stated honestly |
Available is low-cost, high-value (archive the package). Functional/Reusable/Reproduced require the
evaluator's own run to succeed — and for sensor systems, the failure mode is usually "needs hardware
we don't have," so a software/analysis path that reproduces from logged traces is what keeps the
artifact usable when the board is not on the evaluator's bench.
What sensor-systems evaluators open first
| Claim type |
First thing inspected |
Common failure caught |
| An estimator / IP method |
The scripts that turn raw traces into the paper's figures |
Numbers in the PDF that no script reproduces |
| A platform / SPOTS tool |
The firmware build + a run on real or emulated hardware |
Undocumented toolchain; only-builds-on-authors'-bench |
| A dataset |
The raw traces + the extraction/processing scripts + ground truth |
Data shipped without the ground truth or the processing |
| An on-device model |
Firmware + quantized model + a way to run inference |
Requires the exact board and undocumented setup |
| A deployment |
Raw traces + analysis, with a clear "needs the site" statement |
In-field numbers no evaluator can approach |
Assume an evaluator gives your package a bounded time budget and may not have your hardware. Design
the analysis path to succeed on any machine; design the hardware path to be honest about what it
needs.
Packaging plan
[Analysis path] a turnkey script set that regenerates each figure from logged raw traces, on any machine
[Firmware] sources + build instructions + pinned toolchain (compiler/SDK/RTOS) + board revision
[Hardware] BOM / board files, and a clear "you will need X" statement where the board is required
[Data] raw traces + the ground-truth reference (with its error) + calibration data
[Mapping] an explicit table: paper claim -> script/firmware -> expected result
[Harness] the energy/latency measurement setup and conditions, so numbers can be re-measured
[License] an OSI-approved license for code, an open data license for datasets
[Archive] DOI-issuing repository for the Available badge (Zenodo / IEEE DataPort / figshare)
Anonymized review artifact vs. public artifact
- At submission: anonymized for the paper's reviewers — no owner strings, lab-named testbeds,
board silkscreen logos, or identity-revealing dataset DOIs; scrub scope-screenshot watermarks.
- After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing
archive; this is the version evaluators badge, the award judges assess, and the camera-ready cites.
Worked vignette: packaging an on-device sensing artifact
A paper contributes a TinyML detector and a deployment. To target Reusable, Reproduced, and the
award: ship firmware sources with a pinned toolchain and board revision; a reproduce/ directory
whose scripts regenerate every figure from the bundled raw traces on any laptop; the quantized model
and a way to run inference (on the board or an emulator); the labeled field traces with ground-truth
error stated; the power-measurement harness and conditions; an MIT/Apache code license and an open
data license; and a DOI archive. State honestly which results are turnkey (analysis from traces) and
which need the physical board or the deployment site.
Calibration
- The artifact process is typically post-acceptance with its own deadline; do not conflate it
with the camera-ready. Confirm the successor's timing and whether the Best Research Artifact Award
persists (待核实).
- Badge names, the exact set, and whether evaluation is single- or double-anonymous vary by cycle.
Output format
[Target] Available / Functional / Reusable / Reproduced / Best Research Artifact Award
[Artifact role] anonymized review artifact / public DOI-archived artifact
[Contents] <firmware / BOM / traces / ground truth / harness / license>
[Analysis path] does figure regeneration from logged traces succeed on any machine? yes/no
[Hardware honesty] is "you will need X hardware" stated clearly? yes/no
[Claim mapping] <claim -> script/firmware -> expected result present? yes/no>
[Fixes before upload] <ordered list>
1---2name: ipsn-artifact-evaluation3description: Use when packaging an IPSN-lineage artifact for the ACM badges and the IPSN Best Research Artifact Award, covering hardware-plus-software artifacts (firmware, board files, datasets), what sensor-systems evaluators check first, DOI-issuing archives, evaluator-proof documentation for a physical system, and the separate post-acceptance timing.4---56# IPSN Artifact Evaluation78Use this for the artifact track. IPSN had a strong artifact culture — a **Best Research Artifact9Award** alongside the Best Paper Award — and its artifacts are unusual because they mix **hardware,10firmware, and data**, not just code. Two things to internalize: badges and the award are earned by11evaluators actually using your package, and the review artifact (anonymized, for the paper's12reviewers) is not the same deliverable as the public artifact (de-anonymized, DOI-archived). Because13IPSN merged into SenSys, confirm the current badge set, the award's persistence, and the deadline on14the successor call (待核实).1516## Badges and the award (verify the current set)1718| Target | What it certifies | What earns it |19|---|---|---|20| Artifacts Available (ACM) | The artifact is permanently, publicly retrievable | Deposit in a DOI-issuing archive (Zenodo, IEEE DataPort, figshare, Software Heritage) |21| Artifacts Evaluated - Functional | The artifact runs and does what the paper says | A clean-machine (or emulated) install, a demo, documented expected outputs |22| Artifacts Evaluated - Reusable | Others can build on it | The Functional bar plus careful docs, structure, licensing, and board/firmware detail |23| Results Reproduced | An evaluator reproduced key results | A turnkey analysis path from raw traces to the headline numbers |24| IPSN Best Research Artifact Award | Community recognition of an exceptional artifact | A package an evaluator can genuinely run and reuse, hardware caveats stated honestly |2526Available is low-cost, high-value (archive the package). Functional/Reusable/Reproduced require the27evaluator's own run to succeed — and for sensor systems, the failure mode is usually "needs hardware28we don't have," so a **software/analysis path that reproduces from logged traces** is what keeps the29artifact usable when the board is not on the evaluator's bench.3031## What sensor-systems evaluators open first3233| Claim type | First thing inspected | Common failure caught |34|---|---|---|35| An estimator / IP method | The scripts that turn raw traces into the paper's figures | Numbers in the PDF that no script reproduces |36| A platform / SPOTS tool | The firmware build + a run on real or emulated hardware | Undocumented toolchain; only-builds-on-authors'-bench |37| A dataset | The raw traces + the extraction/processing scripts + ground truth | Data shipped without the ground truth or the processing |38| An on-device model | Firmware + quantized model + a way to run inference | Requires the exact board and undocumented setup |39| A deployment | Raw traces + analysis, with a clear "needs the site" statement | In-field numbers no evaluator can approach |4041Assume an evaluator gives your package a bounded time budget and may not have your hardware. Design42the analysis path to succeed on any machine; design the hardware path to be honest about what it43needs.4445## Packaging plan4647```text48[Analysis path] a turnkey script set that regenerates each figure from logged raw traces, on any machine49[Firmware] sources + build instructions + pinned toolchain (compiler/SDK/RTOS) + board revision50[Hardware] BOM / board files, and a clear "you will need X" statement where the board is required51[Data] raw traces + the ground-truth reference (with its error) + calibration data52[Mapping] an explicit table: paper claim -> script/firmware -> expected result53[Harness] the energy/latency measurement setup and conditions, so numbers can be re-measured54[License] an OSI-approved license for code, an open data license for datasets55[Archive] DOI-issuing repository for the Available badge (Zenodo / IEEE DataPort / figshare)56```5758## Anonymized review artifact vs. public artifact5960- **At submission:** anonymized for the paper's reviewers — no owner strings, lab-named testbeds,61 board silkscreen logos, or identity-revealing dataset DOIs; scrub scope-screenshot watermarks.62- **After acceptance:** replace anonymized placeholders with the public, licensed, DOI-issuing63 archive; this is the version evaluators badge, the award judges assess, and the camera-ready cites.6465## Worked vignette: packaging an on-device sensing artifact6667A paper contributes a TinyML detector and a deployment. To target Reusable, Reproduced, and the68award: ship firmware sources with a pinned toolchain and board revision; a `reproduce/` directory69whose scripts regenerate every figure from the bundled raw traces on any laptop; the quantized model70and a way to run inference (on the board or an emulator); the labeled field traces with ground-truth71error stated; the power-measurement harness and conditions; an MIT/Apache code license and an open72data license; and a DOI archive. State honestly which results are turnkey (analysis from traces) and73which need the physical board or the deployment site.7475## Calibration7677- The artifact process is typically **post-acceptance** with its own deadline; do not conflate it78 with the camera-ready. Confirm the successor's timing and whether the Best Research Artifact Award79 persists (待核实).80- Badge names, the exact set, and whether evaluation is single- or double-anonymous vary by cycle.8182## Output format8384```text85[Target] Available / Functional / Reusable / Reproduced / Best Research Artifact Award86[Artifact role] anonymized review artifact / public DOI-archived artifact87[Contents] <firmware / BOM / traces / ground truth / harness / license>88[Analysis path] does figure regeneration from logged traces succeed on any machine? yes/no89[Hardware honesty] is "you will need X hardware" stated clearly? yes/no90[Claim mapping] <claim -> script/firmware -> expected result present? yes/no>91[Fixes before upload] <ordered list>92```
brycewang-stanford