IPSN Experiments
Use this before submission when the evaluation is not yet locked. IPSN reviewers are sensor-systems
and information-processing specialists; the evaluation is where a good idea is won or lost. The
organizing principle is evidence measured on real hardware against real ground truth — the
evaluation must test the sensing claim the paper actually makes, on platforms and baselines a
skeptic would accept.
Evaluation audit
- Measure on real hardware, not just simulation. Simulation can motivate or scale-test, but a
sensing claim needs real sensors: an estimator run on real traces, a pipeline profiled on the
actual MCU, a deployment in a real environment. "Simulation only" is IPSN's classic reject.
- Instrument ground truth. Localization needs surveyed positions; detection needs hand-labeled
events; a physical estimate needs a co-located reference instrument. Report the ground truth's own
error — perfect ground truth is a red flag.
- Measure energy, latency, and footprint on the platform. Report joules/µJ per operation from an
instrumented power rail (name the shunt/instrument/sampling rate), end-to-end latency on the real
SoC at a stated clock, and RAM/flash used vs available. Estimated energy is not measured energy.
- Choose fair, real baselines. Include the strongest prior method and a simple-but-reasonable
alternative (often a classical-DSP or analytic baseline), run under equal conditions on the same
hardware. For IP-track claims, compare to an estimation-theoretic bound (e.g., a Cramér-Rao-style
lower bound) where one exists.
- Isolate the learning's marginal value (on-device/TinyML). Ablate the learned component against
a heuristic/DSP baseline so "did the model help, or the sensing setup?" is answered.
- Design limits in, not on. Know before you deploy which site-specificity, calibration drift, and
generalization limits the study will have, and instrument to bound them.
Claim-to-evidence design table
| Sensing claim |
Matching evidence |
Reject pattern avoided |
| "Estimator is more accurate" |
Error vs ground truth on real traces, with CIs, vs a tuned baseline / a bound |
"Simulated inputs only" |
| "Runs within an energy budget" |
Measured µJ/op on an instrumented rail on the real MCU |
"Energy estimated from datasheet" |
| "Localizes to X meters" |
Surveyed ground-truth positions; error distribution, not just mean |
"Ground truth from the same model being tested" |
| "Deploys reliably" |
Yield, sync error, packet loss over a real deployment duration |
"Idealized single-run numbers" |
| "The on-device model adds value" |
Ablation vs classical DSP / heuristic on the same hardware |
"Model's marginal contribution never isolated" |
| "Scales to N nodes" |
Real or emulated multi-hop at realistic scale, with the bottleneck named |
"Two-node bench test, universal claim" |
On-device / TinyML measurement floor
[Platform] exact MCU/SoC, clock, RAM/flash; the sensor and sampling regime
[Energy] µJ per inference/op, instrument named; duty cycle if always-on
[Latency] end-to-end on-device latency; number of runs and variance
[Footprint] model/pipeline RAM+flash vs available; what had to be quantized/pruned
[Contamination] for learned components, keep train/field data disjoint; report the split
[Ablation] learned component vs DSP/heuristic baseline on the same node
Ground-truth and calibration floor
- Name the ground-truth reference and its own measurement error; a claim can be no better than its
reference.
- Document calibration: procedure, when it was done, and drift over the deployment.
- Archive raw sensor traces and the calibration data, not just derived metrics (see
ipsn-reproducibility).
Deployment reporting floor
- Report yield (fraction of expected data received), synchronization error, packet loss, and energy
over the actual deployment duration, not a best single run.
- State the environment and why it is representative (or not) — external validity for a physical
system is site-bound.
- Report failures: nodes that died, data gaps, and what caused them. Honest deployment reporting is
itself a contribution and a reviewer trust signal.
Vignette: evaluating a localization estimator (IP track)
The paper claims a new estimator localizes better than the prior method. The matching plan: collect
real RF/acoustic traces at surveyed positions; run both estimators on the same traces under equal
tuning; report the full error distribution (not just the mean) with confidence intervals; compare
against the relevant estimation-theoretic bound; and state the environments (indoor/outdoor,
multipath regimes) as a bounded external-validity limit — every number traceable to a logged run and
the surveyed ground truth in the artifact.
Output format
[Evaluation readiness] strong / adequate / weak
[Claim -> evidence map] <claim: platform / ground truth / metric / statistic>
[Real-hardware check] measured on real sensors/MCU, not simulation only? yes/no
[Energy accounting] <µJ/op measured? instrument named? footprint reported?>
[Baseline fairness] <strongest prior + simple baseline, equal conditions, same hardware?>
[Limits-by-design] <site / calibration / generalization -> instrumentation to bound it>
[Decision-critical next run] <one experiment or deployment extension>
Source: brycewang-stanford/Awesome-Journal-Skills → IPSN-Skills/skills/ipsn-experiments/SKILL.md
1---2name: ipsn-experiments3description: Use when designing or auditing IPSN-lineage evaluations, covering real testbeds and deployments, ground-truth instrumentation, energy/latency/footprint measurement on real hardware, on-device/TinyML profiling, estimation-theoretic baselines and bounds, and matching evidence to the shape of each sensing claim across the IP and SPOTS tracks.4---5
6
7# IPSN Experiments
8
9Use this before submission when the evaluation is not yet locked. IPSN reviewers are sensor-systems
10and information-processing specialists; the evaluation is where a good idea is won or lost. The
11organizing principle is **evidence measured on real hardware against real ground truth** — the
12evaluation must test the sensing claim the paper actually makes, on platforms and baselines a
13skeptic would accept.
14
15## Evaluation audit
16
17- **Measure on real hardware, not just simulation.** Simulation can motivate or scale-test, but a
18 sensing claim needs real sensors: an estimator run on real traces, a pipeline profiled on the
19 actual MCU, a deployment in a real environment. "Simulation only" is IPSN's classic reject.
20- **Instrument ground truth.** Localization needs surveyed positions; detection needs hand-labeled
21 events; a physical estimate needs a co-located reference instrument. Report the ground truth's own
22 error — perfect ground truth is a red flag.
23- **Measure energy, latency, and footprint on the platform.** Report joules/µJ per operation from an
24 instrumented power rail (name the shunt/instrument/sampling rate), end-to-end latency on the real
25 SoC at a stated clock, and RAM/flash used vs available. Estimated energy is not measured energy.
26- **Choose fair, real baselines.** Include the strongest prior method and a simple-but-reasonable
27 alternative (often a classical-DSP or analytic baseline), run under equal conditions on the same
28 hardware. For IP-track claims, compare to an estimation-theoretic bound (e.g., a Cramér-Rao-style
29 lower bound) where one exists.
30- **Isolate the learning's marginal value (on-device/TinyML).** Ablate the learned component against
31 a heuristic/DSP baseline so "did the model help, or the sensing setup?" is answered.
32- **Design limits in, not on.** Know before you deploy which site-specificity, calibration drift, and
33 generalization limits the study will have, and instrument to bound them.
34
35## Claim-to-evidence design table
36
37| Sensing claim | Matching evidence | Reject pattern avoided |
38|---|---|---|
39| "Estimator is more accurate" | Error vs ground truth on real traces, with CIs, vs a tuned baseline / a bound | "Simulated inputs only" |
40| "Runs within an energy budget" | Measured µJ/op on an instrumented rail on the real MCU | "Energy estimated from datasheet" |
41| "Localizes to X meters" | Surveyed ground-truth positions; error distribution, not just mean | "Ground truth from the same model being tested" |
42| "Deploys reliably" | Yield, sync error, packet loss over a real deployment duration | "Idealized single-run numbers" |
43| "The on-device model adds value" | Ablation vs classical DSP / heuristic on the same hardware | "Model's marginal contribution never isolated" |
44| "Scales to N nodes" | Real or emulated multi-hop at realistic scale, with the bottleneck named | "Two-node bench test, universal claim" |
45
46## On-device / TinyML measurement floor
47
48```text
49[Platform] exact MCU/SoC, clock, RAM/flash; the sensor and sampling regime
50[Energy] µJ per inference/op, instrument named; duty cycle if always-on
51[Latency] end-to-end on-device latency; number of runs and variance
52[Footprint] model/pipeline RAM+flash vs available; what had to be quantized/pruned
53[Contamination] for learned components, keep train/field data disjoint; report the split
54[Ablation] learned component vs DSP/heuristic baseline on the same node
55```
56
57## Ground-truth and calibration floor
58
59- Name the ground-truth reference and its own measurement error; a claim can be no better than its
60 reference.
61- Document calibration: procedure, when it was done, and drift over the deployment.
62- Archive raw sensor traces and the calibration data, not just derived metrics (see
63 `ipsn-reproducibility`).
64
65## Deployment reporting floor
66
67- Report yield (fraction of expected data received), synchronization error, packet loss, and energy
68 over the *actual* deployment duration, not a best single run.
69- State the environment and why it is representative (or not) — external validity for a physical
70 system is site-bound.
71- Report failures: nodes that died, data gaps, and what caused them. Honest deployment reporting is
72 itself a contribution and a reviewer trust signal.
73
74## Vignette: evaluating a localization estimator (IP track)
75
76The paper claims a new estimator localizes better than the prior method. The matching plan: collect
77real RF/acoustic traces at **surveyed** positions; run both estimators on the same traces under equal
78tuning; report the full error distribution (not just the mean) with confidence intervals; compare
79against the relevant estimation-theoretic bound; and state the environments (indoor/outdoor,
80multipath regimes) as a bounded external-validity limit — every number traceable to a logged run and
81the surveyed ground truth in the artifact.
82
83## Output format
84
85```text
86[Evaluation readiness] strong / adequate / weak
87[Claim -> evidence map] <claim: platform / ground truth / metric / statistic>
88[Real-hardware check] measured on real sensors/MCU, not simulation only? yes/no
89[Energy accounting] <µJ/op measured? instrument named? footprint reported?>
90[Baseline fairness] <strongest prior + simple baseline, equal conditions, same hardware?>
91[Limits-by-design] <site / calibration / generalization -> instrumentation to bound it>
92[Decision-critical next run] <one experiment or deployment extension>
93```
94
95---
96
97**Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `IPSN-Skills/skills/ipsn-experiments/SKILL.md`
thedixitjain