brycewang-stanford SenSys Artifact Evaluation
SenSys awards three independent ACM badges through an Artifact Evaluation Committee: Artifacts Available, Artifacts Evaluated — Functional, and Results Reproduced. They are independent — you may pursue one, two, or all three — and awarded badges are printed on the paper and recorded in the ACM DL. The hard part at SenSys is that your evidence is physical: an evaluator usually does not have your motes, your harvester, or your deployment, so the artifact must be built to be judged without them.
Choose the badges deliberately
| Badge | Bar | Hardest part at SenSys |
|---|---|---|
| Artifacts Available | Artifact deposited in a permanent public archive with a DOI | Deciding what firmware/traces you can legally and safely release |
| Artifacts Evaluated — Functional | The artifact runs and does what the paper says | Giving an evaluator without your hardware a way to reach "it runs" |
| Results Reproduced | Key results re-obtained by the evaluator | Reproducing hardware-measured energy/latency numbers off your testbed |
Available is the cheapest and worth claiming almost always. Functional and Reproduced are where the hardware-optional path earns its keep.
Build the hardware-optional path first
Most AEC members will not have your testbed. Give them a graded way in:
Tier 0 — Available: archived repo (DOI), firmware sources, traces, README.
Tier 1 — Bench/replay: recorded sensor + power traces the analysis re-runs on any laptop,
reproducing the paper's figures from stored data.
Tier 2 — Emulation: a QEMU/renode-class emulator or a single dev board that reaches
"it runs" without the full deployment.
Tier 3 — Full HW: scripts + BOM for an evaluator who does have the platform.
A trace-replay path is the single most valuable thing you can ship: it lets an evaluator reproduce your figures from your recorded energy and sensor data even if they cannot re-run the deployment. Document exactly which figures/tables the replay reproduces and which need hardware.
Document the provenance the AEC cannot infer
artifact/
README.md # claims → which script/trace reproduces each figure/table
firmware/ # sources + toolchain version + build flags
traces/ # recorded power + sensor data behind each figure
analysis/ # scripts that turn traces into the paper's plots
hardware/ # BOM, board revision, wiring, instrument model + settings
ENERGY.md # instrument, sampling rate, integration boundaries
GROUND_TRUTH.md # how reference labels were obtained and their error
LICENSE
The ENERGY.md and GROUND_TRUTH.md files are SenSys-specific: an evaluator reproducing a
number needs the measurement method, not just the code (see sensys-reproducibility).
Smoke-run before you submit
Prove the package installs and runs from clean before an evaluator ever sees it:
# Package hygiene (manifests, seeds, scripts) — see resources/code/README.md
python3 ../../../shared-resources/ml-conference-methods/code/check_repro_package.py ./artifact
# Then the SenSys-specific smoke: does the trace-replay reproduce a headline figure?
cd artifact/analysis && ./reproduce_fig4.sh # should regenerate Fig. 4 from traces/, no hardware
If the replay does not regenerate a figure on a clean machine, no badge claim is safe yet.
Work with the AEC's iteration
The committee may ask for revisions and iterate with you. Respond fast and concretely: a missing dependency or an unclear step is a quick fix, and the AEC is trying to award the badge, not deny it. Keep the anonymity rules of the round if evaluation overlaps the review window.
Output format
[Badges] which of the three you are pursuing, and why each is reachable
[HW-path] the graded path (Available/Replay/Emulation/Full-HW) — which tiers exist
[Provenance] ENERGY.md + GROUND_TRUTH.md + firmware toolchain present? pass/gap
[Smoke] does trace-replay reproduce a headline figure on a clean machine? Y/N
[Open] the gap most likely to block Functional or Reproduced