brycewang-stanford SenSys Experiments
At SenSys the evaluation is the contribution's proof. A mechanism is only as strong as the measurements that show it behaving on real hardware, under real energy budgets, against honest ground truth. This skill audits an evaluation for the failure modes SenSys reviewers flag first: unmeasured energy, simulation standing in for deployment, and accuracy scored against an unstated truth.
The five measurement axes
Every SenSys evaluation should be explicit about these; a gap in any one is a reviewer question.
| Axis | What to report | Common failure |
|---|---|---|
| Energy / power | Energy per operation, average current (µA/mA), duty cycle — with instrument + sampling rate + wake/sleep boundaries | "Low-power" as an adjective; no method |
| Latency | On-device latency as a distribution (median, tail), not a single number | One workstation timing, no tail |
| Accuracy vs. ground truth | Metric plus how truth was obtained and its own error | Accuracy with unstated reference |
| Deployment realism | Node count, placement, environment, duration, uptime/failures | One-run bench result called a deployment |
| Footprint (embedded AI) | Quantized model size, RAM/flash peak on the actual MCU | Offline model size on a workstation |
Measure energy like it is the headline
Because it is. State the instrument (source-meter, shunt + DAQ, or power monitor), its sampling rate, and the boundaries of what you integrated (does "energy per inference" include sensor acquisition and radio, or only compute?). Report energy with the same rigor as a latency CDF — ideally a power trace annotated with the phases it covers.
Energy reporting template (put the method in the paper, not just the number):
Instrument: Keithley/Otii/INA-class monitor, model + firmware
Sampling rate: e.g. 10 kHz; enough to resolve the wake spike
Integration: from sensor-on to label-out; radio TX included? Y/N
Boundaries: sleep floor measured separately; not inferred from datasheet
Report: energy/op + duty cycle + projected lifetime with battery/harvester spec
A projected battery life or harvesting budget must be derived from measured draw, not from a datasheet's nominal current — reviewers who have deployed will catch the difference.
Ground truth is a first-class artifact
An accuracy number is only as trustworthy as the truth it is scored against. State how reference labels were obtained — a reference instrument, a controlled stimulus, or a documented human annotation protocol — and the truth's own uncertainty. Sensor experiments where "ground truth" is another uncalibrated sensor, or where labels were assigned by the authors without a protocol, invite exactly the challenge that sinks the result in review.
Deployment vs. bench, kept separate
Bench experiments control variables to isolate a mechanism; deployments expose it to reality. Report both and label which is which. A deployment carries node count, placement, environmental conditions, duration, and honest uptime/failure accounting — a node that died on day 3 is data, not an embarrassment to hide. A single controlled bench run is not a deployment claim.
Baselines on the same hardware
Compare against the right prior system on the same platform, tuned as well as your own. An apples-to-oranges comparison — your system on an optimized MCU against a baseline you ran untuned, or against numbers copied from a paper that used different silicon — is the most common reviewer objection. If you must cite cross-hardware numbers, say so and bound the comparison.
Intermittent power and harvesting
For batteryless or energy-harvesting systems, the evaluation must include the energy source's
behavior: the harvest trace (indoor light, RF, vibration), the capacitor/energy-buffer sizing,
and behavior across power failures. A harvesting claim without the input-energy conditions is not
reproducible even with the code (see sensys-reproducibility).
Audit checklist
[ ] Energy reported with instrument + sampling rate + integration boundaries.
[ ] Latency as a distribution (median + tail), measured on the target device.
[ ] Accuracy paired with ground-truth provenance and the truth's own error.
[ ] Deployment: node count, environment, duration, uptime/failures stated.
[ ] Baselines run on the same hardware, tuned; cross-hardware numbers flagged.
[ ] Embedded-AI: quantized size + RAM/flash peak on the actual MCU.
[ ] Harvesting: input-energy trace + buffer sizing + power-failure behavior.
[ ] No simulation-only or single-run claim standing in for deployed behavior.
Output format
[Axes] which of the five measurement axes are covered / missing
[Energy] method stated? instrument + sampling rate + boundaries — pass/gap
[Truth] ground-truth provenance and its error — pass/gap
[Deploy] deployment realism + honest uptime — pass/gap
[Baseline] same-hardware, tuned comparison — pass/gap
[Open] the single measurement whose absence most weakens the paper