brycewang-stanford SenSys Reproducibility
A SenSys result is reproducible when someone on different hardware can understand — and, from your traces, re-obtain — your numbers. The threat is not messy code; it is provenance that evaporates when the deployment ends. Energy measured without its method, accuracy scored against forgotten ground truth, and firmware whose toolchain is unrecorded cannot be reproduced at any price once the testbed is torn down. Capture it while the nodes are still powered.
Capture it live, not later
| Provenance | Capture while live | Why it cannot be reconstructed |
|---|---|---|
| Energy method | Instrument model, sampling rate, integration boundaries, sleep floor | A stored current number loses its method; "42 µA" of what phases? |
| Hardware | MCU/SoC part + clock, sensor config, radio + TX power, board revision, battery/harvester spec | A later "same board" is rarely bit-identical |
| Firmware | Sources + toolchain version + compiler flags, pinned | A rebuild on a new toolchain shifts timing and energy |
| Ground truth | Reference protocol + the reference's own error | Labels without their protocol are unfalsifiable |
| Deployment | Placement, environment, duration, node uptime/failures | Conditions are gone once the deployment ends |
| Harvest input | Energy-source trace (light/RF/vibration), buffer sizing | Behavior across brownouts is unreproducible without the input |
The reproducibility that matters is cross-hardware
Reproducing your own result on your own testbed proves little. The SenSys bar is that a
different lab can interpret a mismatch: when their number differs from yours, the provenance
tells them why (different MCU clock, different harvest input) rather than leaving them to guess.
Ship recorded traces so the analysis can be re-run even by someone who lacks your hardware —
this is what turns a deployment result into a reproducible one and underpins the trace-replay
path in sensys-artifact-evaluation.
Minimum reproducible bundle for a SenSys figure:
traces/fig4_power.csv # raw power samples + timestamps
traces/fig4_sensor.csv # the sensor stream behind the same figure
analysis/reproduce_fig4.py # regenerates Fig. 4 from the two traces
ENERGY.md # instrument, rate, integration boundaries
HARDWARE.md # part numbers, clocks, board rev, harvester spec
firmware/ + toolchain.lock # pinned build that produced the traces
Decide early what can ship
Some SenSys artifacts carry release constraints that a purely computational paper never faces:
- Sensor data of people or spaces may need consent, anonymization, or aggregation before it can be released — decide at collection time, because retroactive consent is usually impossible.
- Firmware and hardware designs may touch third-party IP or a vendor NDA; confirm what is releasable before promising an Available badge.
- Deployment locations can be sensitive (critical infrastructure, private property); scrub identifying detail from released traces.
Resolving these late forces a choice between a weak artifact and a broken promise. Resolve them in
the experiment plan (sensys-experiments).
Reproducibility checklist
[ ] Energy method (instrument, rate, boundaries) recorded, not just the number.
[ ] Hardware provenance: parts, clocks, board revision, battery/harvester spec.
[ ] Firmware sources + pinned toolchain + compiler flags archived.
[ ] Ground-truth protocol and the reference's own error documented.
[ ] Deployment conditions + honest uptime/failure log captured while live.
[ ] Harvest-input traces + buffer sizing stored for batteryless results.
[ ] Recorded traces shipped so figures re-generate without your hardware.
[ ] Release constraints (consent, NDA, location) resolved at collection time.
Output format
[Live-capture] which provenance is captured vs. still at risk while the testbed runs
[Cross-HW] can a different lab interpret a mismatch from what you shipped? Y/N
[Traces] do shipped traces regenerate the headline figures without hardware? Y/N
[Release] consent/NDA/location constraints resolved? open items
[Open] the provenance whose loss would most damage reproducibility