Mobisys Experiments

MobiSys Experiments

A MobiSys result becomes believable only when it is measured on real hardware, driven to steady state, and reported in latency, energy, memory, and thermal terms. Simulation-only or single-run numbers read as under-done. The evaluation axis here is the device and platform — compute, energy, latency, memory, heat — not the radio or channel a wireless venue grades, and not the traffic-and-topology an infrastructure venue grades. Build the evaluation as one device-behavior claim per experiment.

Turn each claim into one device experiment

The clean structure gives each subsection a claim about the running system, then the one experiment that settles it. The opposite habit — run a benchmark suite and describe whatever survives — is the unfocused evaluation MobiSys reviewers push back on. The claim-to-experiment map for a running mobile system:

What the setup must state

• The full device context: phone/board model, SoC, OS build, framework/runtime version, battery vs. wall power, and ambient temperature. A number without its device is not interpretable.
• Define the energy boundary: the power instrument (on-device rail, external monitor, shunt + DAQ), the sampling rate, and what is inside the measured envelope (SoC only? whole device?). Do not let "efficient" stand in for a measured joules figure.
• Make load explicit: the workload trace, its duration, and whether the run reached thermal steady state; a 30-second benchmark hides the throttling that a 20-minute run shows.
• Report ground truth honestly: for a service or inference claim, how correctness was established and its own error.

Making energy and heat auditable

Battery, energy, and thermal claims are common at MobiSys and are held to instrument-level scrutiny:

Energy/thermal report checklist:
  instrument: power monitor / on-device rail / shunt + DAQ, sampling rate
  quantity: energy-per-operation (mJ), not just average power (mW)
  boundary: what is inside the measured envelope (SoC? whole device?)
  thermal: skin/SoC temperature trace and whether steady state was reached
  battery: state-of-charge span, or wall power stated explicitly

An energy or thermal claim that cannot be re-derived from a described setup should not survive your own audit.

Comparators a systems reviewer will trust

• Compare against the system people actually run, configured the way its own documentation prescribes — an untuned baseline is the most common credibility wound in systems reviewing.
• Include the do-less baseline: the trivial fix (a smaller model, a fixed low frame rate, more aggressive quantization). If the system cannot beat it at equal quality, that is the finding.
• When a competitor cannot be run (proprietary runtime, unavailable device), reimplement and label it a reimplementation, or compare on published numbers with the configuration deltas stated.

Report the spread, never a lone peak

• Report percentiles and confidence intervals for latency, energy, and throughput; a single "up to N×" without the distribution is a review risk (mobisys-writing-style).
• Show tails and CDFs for headline latency results, and repeat runs across enough devices, thermal states, or battery levels to expose run-to-run spread.
• On-device results are device- and state-dependent — a number from one phone at full charge in a cool room is not a claim about the system until the spread is characterized.

Pre-submission evaluation pass

• Every evaluation subsection answers a named question.
• Device, OS, runtime version, power source, and ambient fully specified per experiment.
• Energy measured with a described instrument and boundary, not estimated.
• Sustained-load run reaches steady state; thermal behavior reported.
• Incumbent-grade baseline present and tuned; do-less baseline present.
• Percentiles + tails for headline latency; multi-device or multi-state spread reported.
• At least one device state the system does not win, discussed rather than buried.
• Numbers in abstract/intro regenerate from the recorded runs.
• A hardware-optional or downscaled variant exists for artifact evaluators (mobisys-artifact-evaluation).

Output format

[Evidence form] on-device measurement / deployment / user study / trace / simulation (claimed vs actual)
[Question map] question -> experiment -> metric (gaps flagged)
[Device+energy] setup and boundary specified? y/n per headline experiment
[Sustained load] steady state reached and thermal reported? y/n
[Baseline audit] incumbent tuned? do-less present?
[Break condition] device state where the system loses: <named or MISSING>
[Priority additions] ordered by review-risk reduction per bench-day

Source: brycewang-stanford/Awesome-Journal-Skills → MobiSys-Skills/skills/mobisys-experiments/SKILL.md