MobiSys Reproducibility
Use this before submission and again before the artifact-evaluation deadline. On-device
results are fragile across hardware and framework versions, so reproducibility at MobiSys is
mostly about provenance: recording enough of the device and measurement context that
someone on a different phone can rebuild the result.
Evidence map
- Map each latency, energy, throughput, and accuracy claim to a verifiable location in the
paper, appendix, or artifact package.
- For each on-device result, record device model, SoC, OS build, framework/runtime version,
model checkpoint hash, power source, ambient temperature, and thermal state at run start.
- Record the power-instrument setup and the energy boundary so an energy figure can be
re-derived rather than trusted.
- For stochastic results, report seeds, repeated runs, and the spread; a single number from a
single phone is not reproducible evidence.
- Explain missing data, firmware, or hardware honestly, and describe how a reader could
reproduce the result in principle on a different device.
- Keep the artifact consistent with the manuscript; a number in the PDF that the artifact
cannot regenerate is a review-risk multiplier and an AEC failure.
What survives a different device
| Provenance item |
Turnkey answer |
Common failure caught |
| Device + SoC + OS build |
Named per experiment, in a table |
"on a mobile device" with no model |
| Framework / runtime version |
Pinned with hashes |
Result drifts silently across a minor version |
| Energy instrument + boundary |
Instrument, sampling rate, envelope stated |
"efficient" with no measured joules |
| Thermal state |
Ambient + steady-state trace |
30-second benchmark hides throttling |
| Model / cache version |
Checkpoint hash, cache size |
Accuracy irreproducible from a different weight file |
Marking a device result "reproducible" without its provenance is a recognizable MobiSys red
flag, because the AEC and reviewers cross-check the artifact against the PDF and read a gap as
carelessness about the rest of the paper.
Vignette: an on-device inference paper
Consider a runtime that keeps CNN latency stable under load, validated on four phones. Its
reproducibility spine: the four device models and OS builds, the runtime version and model
checkpoint hashes, the power-monitor model and sampling rate, the ambient temperature and the
steady-state thermal traces, the workload trace, the seeds and run counts — plus one honest
sentence about the device it could not test and why the result may differ there.
Degrees of reproducibility
- Turnkey: one script regenerates each figure from logged runs on the target device.
- Scripted: scripts exist but need the specific hardware or manual device setup.
- Descriptive: prose detailed enough that a competent reader could rebuild the pipeline on
a comparable device.
For MobiSys, the on-device measurement path should be as turnkey as the hardware allows, and a
hardware-optional downscaled or emulator path should let an evaluator without your exact
device reach at least Functional (mobisys-artifact-evaluation). State the achieved level
honestly rather than overpromising turnkey behavior that fails on a different phone.
Legal and privacy provenance
- Decide early what traces, firmware, and datasets can ship; on-device data often carries
user, sensor, or vendor-firmware restrictions.
- Strip identifying content from screenshots, logs, and demo media before release.
- If data cannot ship, provide a synthetic or public substitute and name which results it
cannot reproduce.
Output format
[Claim inventory] <claim -> evidence location>
[Provenance status] complete / partial / missing (device/OS/runtime/energy/thermal)
[Reproducibility gaps] <device pinning / seeds / energy boundary / model version>
[Paper fixes] <must appear in main PDF>
[Artifact fixes] <appendix or artifact additions, incl. hardware-optional path>
1---2name: mobisys-reproducibility3description: Use when strengthening MobiSys reproducibility evidence — capturing device, SoC, OS build, framework and model versions, power-instrument setup, seeds, and thermal conditions so an on-device result survives a different phone, deciding what data and firmware can legally ship, and keeping the paper consistent with the artifact for the ACM badge pipeline.4---56# MobiSys Reproducibility78Use this before submission and again before the artifact-evaluation deadline. On-device9results are fragile across hardware and framework versions, so reproducibility at MobiSys is10mostly about **provenance**: recording enough of the device and measurement context that11someone on a different phone can rebuild the result.1213## Evidence map1415- Map each latency, energy, throughput, and accuracy claim to a verifiable location in the16 paper, appendix, or artifact package.17- For each on-device result, record device model, SoC, OS build, framework/runtime version,18 model checkpoint hash, power source, ambient temperature, and thermal state at run start.19- Record the power-instrument setup and the energy boundary so an energy figure can be20 re-derived rather than trusted.21- For stochastic results, report seeds, repeated runs, and the spread; a single number from a22 single phone is not reproducible evidence.23- Explain missing data, firmware, or hardware honestly, and describe how a reader could24 reproduce the result in principle on a different device.25- Keep the artifact consistent with the manuscript; a number in the PDF that the artifact26 cannot regenerate is a review-risk multiplier and an AEC failure.2728## What survives a different device2930| Provenance item | Turnkey answer | Common failure caught |31|---|---|---|32| Device + SoC + OS build | Named per experiment, in a table | "on a mobile device" with no model |33| Framework / runtime version | Pinned with hashes | Result drifts silently across a minor version |34| Energy instrument + boundary | Instrument, sampling rate, envelope stated | "efficient" with no measured joules |35| Thermal state | Ambient + steady-state trace | 30-second benchmark hides throttling |36| Model / cache version | Checkpoint hash, cache size | Accuracy irreproducible from a different weight file |3738Marking a device result "reproducible" without its provenance is a recognizable MobiSys red39flag, because the AEC and reviewers cross-check the artifact against the PDF and read a gap as40carelessness about the rest of the paper.4142## Vignette: an on-device inference paper4344Consider a runtime that keeps CNN latency stable under load, validated on four phones. Its45reproducibility spine: the four device models and OS builds, the runtime version and model46checkpoint hashes, the power-monitor model and sampling rate, the ambient temperature and the47steady-state thermal traces, the workload trace, the seeds and run counts — plus one honest48sentence about the device it could *not* test and why the result may differ there.4950## Degrees of reproducibility5152- **Turnkey:** one script regenerates each figure from logged runs on the target device.53- **Scripted:** scripts exist but need the specific hardware or manual device setup.54- **Descriptive:** prose detailed enough that a competent reader could rebuild the pipeline on55 a comparable device.5657For MobiSys, the on-device measurement path should be as turnkey as the hardware allows, and a58**hardware-optional** downscaled or emulator path should let an evaluator without your exact59device reach at least *Functional* (`mobisys-artifact-evaluation`). State the achieved level60honestly rather than overpromising turnkey behavior that fails on a different phone.6162## Legal and privacy provenance6364- Decide early what **traces, firmware, and datasets** can ship; on-device data often carries65 user, sensor, or vendor-firmware restrictions.66- Strip identifying content from screenshots, logs, and demo media before release.67- If data cannot ship, provide a synthetic or public substitute and name which results it68 cannot reproduce.6970## Output format7172```text73[Claim inventory] <claim -> evidence location>74[Provenance status] complete / partial / missing (device/OS/runtime/energy/thermal)75[Reproducibility gaps] <device pinning / seeds / energy boundary / model version>76[Paper fixes] <must appear in main PDF>77[Artifact fixes] <appendix or artifact additions, incl. hardware-optional path>78```
brycewang-stanford