brycewang-stanford MobiCom Reproducibility
Reproducibility at MobiCom is harder than at a compute-only venue because the experiment includes a physical radio channel that no one else has. The goal is not bit-identical reruns; it is enough provenance that a reader with a different testbed can tell whether a mismatch is their setup or your claim. Provenance is captured while the runs happen — it cannot be reconstructed after the testbed is torn down.
The provenance ledger
Keep a machine-readable record per experiment, written as the run executes:
Per-run provenance record:
radio: SDR/NIC model, firmware/driver version, carrier freq, bandwidth, TX power
rf_frontend: antenna type, gain, cabling, amplifiers
topology: node count, positions/distances, LOS/NLOS, host specs
channel: RSSI/CSI traces, measured interference, coherence-time estimate
mobility: walker paths, speeds, schedule (or "stationary")
energy: instrument, sampling rate, measured boundary
software: code commit hash, config diffs from defaults, analysis-script hash
outputs: raw-capture location, per-figure derivation script
This ledger is simultaneously the evaluation record (mobicom-experiments), the
artifact-evaluation seed (mobicom-artifact-evaluation), and the insurance policy for a
one-shot revision that asks you to re-run months later on the same setup
(mobicom-author-response).
Characterize the variance, do not hide it
Over-the-air numbers move between runs. A reproducible MobiCom result states its spread:
| Source of variance | How to characterize it |
|---|---|
| Channel realization | repeat across times of day / room states; report the distribution |
| Interference | log the ambient band occupancy during runs; note co-channel activity |
| Hardware unit | test more than one radio/tag where feasible; note per-unit drift |
| Mobility | multiple walker paths and speeds; report per-path spread |
| Thermal / time | long runs to expose drift; report start-vs-end |
If a headline figure comes from one favorable channel realization, it is not yet a claim about the mechanism — say what varied and report the range, not the best run.
What can legally and safely ship
Decide early, because it constrains both the artifact and the anonymity sweep:
- Raw captures and traces: confirm the collection did not record identifiable users or protected content; strip or aggregate what cannot ship. IRB or site conditions may bind release.
- Firmware and proprietary radio stacks: often not redistributable — plan a description and a substitute rather than a broken artifact.
- Deployment-site data: building maps, floor plans, or site identifiers can both
de-anonymize (
mobicom-submission) and violate a site agreement; generalize them. - Third-party datasets: ship a loader and instructions, not a re-hosted copy, unless the license permits.
A hardware-optional reproduction path
Most readers and most artifact evaluators will not have your radios. Plan, from the start, a path that does not require them:
- A trace-replay mode that feeds recorded captures through the same decode/analysis pipeline, so the processing is reproducible without a testbed.
- A downscaled configuration (fewer nodes, shorter distance) an evaluator can run on common hardware to reach a Functional result.
- Expected-output and expected-variance files, so a rerun that differs by a known margin reads as success, not failure.
Audit checklist
- Provenance ledger captured per run, not reconstructed.
- RF, channel, and mobility conditions recorded with each headline result.
- Variance sources characterized; ranges reported, not single best runs.
- Legal/ethical shipping decision made for traces, firmware, and site data.
- Trace-replay or downscaled path exists for testbed-less reproduction.
- Analysis scripts version-pinned to the raw captures they consume.
Output format
[Provenance] ledger present per run? gaps named
[Variance] which sources characterized; which unmeasured
[Shipping] traces / firmware / site data — ship, strip, or substitute (each)
[Testbed-less path] trace-replay / downscaled present? y/n
[Risks] the reruns most likely to diverge, and why