brycewang-stanford MobiSys Artifact Evaluation
Use this for the MobiSys artifact submission. MobiSys has a strong, script-first artifact culture: the single-blind AEC runs the authors' workflow scripts on the reviewers' own systems, so an artifact that only works on your bench is not an artifact.
The three ACM badges
Authors may seek one, two, or all three independent badges; passing badges are printed on the paper and recorded in the ACM DL:
| Badge | What it certifies | What the AEC does |
|---|---|---|
| Artifacts Available | The artifact is publicly archived at a stable location | Checks the archival link and license |
| Artifacts Evaluated — Functional | The artifact runs and does what the paper says | Runs your workflow scripts on the AEC's machines |
| Results Reproduced | Key paper results are reproduced independently | Regenerates headline numbers/figures from your package |
Choose badges as claim calibration: Available is a release commitment; Functional needs a runnable package; Results Reproduced needs the headline device results to come back on machines you do not control — the hardest badge for on-device work.
Artifact plan
- Decide what evidence supports each badge: source, build scripts, model files, workload traces, and the measurement/plotting pipeline.
- Provide a top-level workflow script and a README that orients an evaluator in one minute: environment, dependencies, one command to smoke-test, one to reproduce a headline figure, expected outputs, and runtime.
- Anonymize repository history, paths, screenshots, notebook metadata, license headers, and device serial numbers for the review round; de-anonymize only at camera-ready.
- Pin framework, runtime, and model checkpoint hashes; on-device results drift across framework minor versions.
The hardware-optional path (the MobiSys-specific move)
On-device artifacts fail Functional most often because the evaluator lacks the exact phone, wearable, or board. Design for that from the start:
Hardware-optional package layout:
/device full pipeline for the exact hardware (Results Reproduced target)
/emulated downscaled or emulator path an evaluator can run without the device
/logs recorded runs so plots regenerate even if nothing executes
/README states which claims each path can and cannot reproduce
State plainly which headline numbers the emulated path cannot reproduce (typically absolute energy and thermal behavior) so the AEC grades the right badge instead of failing the whole package.
What AEC reviewers open first
- The README and the top-level workflow script — if these do not run cleanly, the badge stalls regardless of how good the system is.
- The one-command smoke test — make it fast and hermetic.
- The plotting pipeline for a headline figure — emit figures directly from logged results so the paper numbers and artifact numbers cannot drift.
Calibration anchors
- Artifact evaluation is single-blind and separate from paper review; the badge names and the artifact deadline are cycle-specific — verify against the current Artifact Evaluation page rather than past years.
- Absolute on-device energy and latency are the least portable results; scope Results Reproduced claims to what survives on another evaluator's hardware.
Output format
[Badges sought] Available / Functional / Results Reproduced
[Package contents] <source/scripts/models/traces/plotting>
[Hardware-optional path] present? which claims it cannot reproduce
[Anonymity risks] <paths/screenshots/serials/repo owners>
[Reproduction level] turnkey / scripted / descriptive / weak
[Fixes before upload] <ordered list>