# Mobisys Experiments > Use when designing or auditing the evaluation of a MobiSys submission — building real-device testbeds, instrumenting energy and thermal behavior, measuring latency and frame-rate tails, bounding memory footprint, choosing tuned system baselines, and running deployments or user studies, so systems reviewers see where the system wins and breaks on the device. - Skill: `thedixitjain/mobisys-experiments` (Agent Skill) - Install (CLI): `npx skillmds add thedixitjain/mobisys-experiments` - Raw SKILL.md: https://api.skillmd.com/api/skills/thedixitjain/mobisys-experiments/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: thedixitjain (https://skillmd.com/u/thedixitjain) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/thedixitjain/mobisys-experiments --- # MobiSys Experiments A MobiSys result becomes believable only when it is **measured on real hardware, driven to steady state, and reported in latency, energy, memory, and thermal terms**. Simulation-only or single-run numbers read as under-done. The evaluation axis here is the *device and platform* — compute, energy, latency, memory, heat — not the radio or channel a wireless venue grades, and not the traffic-and-topology an infrastructure venue grades. Build the evaluation as one device-behavior claim per experiment. ## Turn each claim into one device experiment The clean structure gives each subsection a claim about the running system, then the one experiment that settles it. The opposite habit — run a benchmark suite and describe whatever survives — is the unfocused evaluation MobiSys reviewers push back on. The claim-to-experiment map for a running mobile system: | Claim about the system | Experiment that settles it | Metrics | |---|---|---| | It meets the target in the motivating scenario | on-device run under the real workload | p50/p95 latency, throughput, task success | | Its energy cost is acceptable | instrumented run on a power monitor | energy-per-operation, average and peak power | | It survives sustained use | multi-minute run to thermal steady state | frame-rate stability, temperature trace, throttle onset | | Each mechanism earns its place | component breakdown / ablation | per-mechanism contribution | | It fits within device memory | peak-memory and storage accounting | peak RSS, model and cache size | | Its failure mode is understood | hot, low-memory, low-battery device states | the state where a baseline wins | ## What the setup must state - **The full device context:** phone/board model, SoC, OS build, framework/runtime version, battery vs. wall power, and ambient temperature. A number without its device is not interpretable. - **Define the energy boundary:** the power instrument (on-device rail, external monitor, shunt + DAQ), the sampling rate, and what is inside the measured envelope (SoC only? whole device?). Do not let "efficient" stand in for a measured joules figure. - **Make load explicit:** the workload trace, its duration, and whether the run reached thermal steady state; a 30-second benchmark hides the throttling that a 20-minute run shows. - **Report ground truth honestly:** for a service or inference claim, how correctness was established and its own error. ## Making energy and heat auditable Battery, energy, and thermal claims are common at MobiSys and are held to instrument-level scrutiny: ```text Energy/thermal report checklist: instrument: power monitor / on-device rail / shunt + DAQ, sampling rate quantity: energy-per-operation (mJ), not just average power (mW) boundary: what is inside the measured envelope (SoC? whole device?) thermal: skin/SoC temperature trace and whether steady state was reached battery: state-of-charge span, or wall power stated explicitly ``` An energy or thermal claim that cannot be re-derived from a described setup should not survive your own audit. ## Comparators a systems reviewer will trust - Compare against the **system people actually run**, configured the way its own documentation prescribes — an untuned baseline is the most common credibility wound in systems reviewing. - Include the **do-less** baseline: the trivial fix (a smaller model, a fixed low frame rate, more aggressive quantization). If the system cannot beat it at equal quality, that is the finding. - When a competitor cannot be run (proprietary runtime, unavailable device), reimplement and label it a reimplementation, or compare on published numbers with the configuration deltas stated. ## Report the spread, never a lone peak - Report **percentiles and confidence intervals** for latency, energy, and throughput; a single "up to N×" without the distribution is a review risk (`mobisys-writing-style`). - Show **tails and CDFs** for headline latency results, and repeat runs across enough devices, thermal states, or battery levels to expose run-to-run spread. - On-device results are **device- and state-dependent** — a number from one phone at full charge in a cool room is not a claim about the system until the spread is characterized. ## Pre-submission evaluation pass - [ ] Every evaluation subsection answers a named question. - [ ] Device, OS, runtime version, power source, and ambient fully specified per experiment. - [ ] Energy measured with a described instrument and boundary, not estimated. - [ ] Sustained-load run reaches steady state; thermal behavior reported. - [ ] Incumbent-grade baseline present and tuned; do-less baseline present. - [ ] Percentiles + tails for headline latency; multi-device or multi-state spread reported. - [ ] At least one device state the system does not win, discussed rather than buried. - [ ] Numbers in abstract/intro regenerate from the recorded runs. - [ ] A hardware-optional or downscaled variant exists for artifact evaluators (`mobisys-artifact-evaluation`). ## Output format ```text [Evidence form] on-device measurement / deployment / user study / trace / simulation (claimed vs actual) [Question map] question -> experiment -> metric (gaps flagged) [Device+energy] setup and boundary specified? y/n per headline experiment [Sustained load] steady state reached and thermal reported? y/n [Baseline audit] incumbent tuned? do-less present? [Break condition] device state where the system loses: [Priority additions] ordered by review-risk reduction per bench-day ``` --- **Source:** [`brycewang-stanford/Awesome-Journal-Skills`](https://github.com/brycewang-stanford/Awesome-Journal-Skills) → `MobiSys-Skills/skills/mobisys-experiments/SKILL.md`