Mobile Inference Benchmark Eval

mobile-inference-benchmark-eval

Cloud-based or On-device: An Empirical Study of Mobile Deep Inference — Guo (2017) (arXiv:1707.04610, 2017)

What this evaluates

Evaluates the practical feasibility of running deep learning models on mobile hardware by measuring end-to-end latency and energy consumption during CNN inference. It compares on-device execution against cloud-based execution to identify hardware and network bottlenecks.

Datasets

• Mobile Benchmark Image Set — total 15; splits: test (15)

Metrics

• end-to-end latency (primary) — range: other
  • Measures the total time from image input to receiving the top-5 label output, including model loading, computation, and network transfer (for cloud mode).
• energy consumption — range: other
  • Total power draw measured during the inference process, capturing overhead from model loading and computation.

Input / output format

Input: 224x224 pixel image bitmaps (original or downscaled).

Output: Probability distribution over labels, specifically the top 5 most probable labels.

Scoring recipe

for each image in test_set:
  start_t = time.perf_counter()
  start_e = measure_power()
  output = run_inference(image) # on-device or cloud
  end_t = time.perf_counter()
  end_e = measure_power()
  latencies.append(end_t - start_t)
  energies.append(end_e - start_e)
avg_latency = sum(latencies) / len(latencies)
avg_energy = sum(energies) / len(energies)

Common pitfalls

• Network jitter and server load can significantly skew cloud-based latency measurements.
• Model loading overhead often dominates on-device inference time, masking actual computation speed.
• Framework choice (CPU-only Caffe vs. GPU-enabled CNNDroid) drastically changes on-device performance baselines.

Evidence (verbatim from paper)

An empirical evaluation of on-device vs. cloud-based deep inference on mobile devices reveals that on-device CNN inference incurs up to two orders of magnitude higher end-to-end latency and energy consumption due to model loading and computation bottlenecks.

Citation

@misc{guo2017mobileinference,
  title={Cloud-based or On-device: An Empirical Study of Mobile Deep Inference},
  author={Guo (2017)},
  year={2017},
  note={arXiv:1707.04610}
}

• arXiv: 1707.04610