Phonebit Inference Bench

phonebit-inference-bench

PhoneBit: Efficient GPU-Accelerated Binary Neural Network Inference Engine for Mobile Phones — Chen et al. (2019) (arXiv:1912.04050, 2019)

What this evaluates

Evaluates the inference speed, energy efficiency, and classification accuracy of a GPU-accelerated binary neural network engine on mobile devices against standard frameworks.

Datasets

• CIFAR10 — total ?; splits: test (-1)
• VOC2007 — total ?; splits: test (-1)

Metrics

• accuracy — range: percent
  • Percentage of correctly classified images in the dataset.
• model size — range: other
  • Total storage footprint of the neural network parameters in megabytes.
• runtime (primary) — range: other
  • Average inference time per image in milliseconds across the dataset.
• power — range: other
  • Average power draw during inference measured in milliwatts.
• energy efficiency — range: other
  • Frames processed per watt of energy consumed, calculated as FPS divided by power draw.

Input / output format

Input: Pre-trained binary and full-precision CNN models (AlexNet, YOLOv2 Tiny, VGG16) deployed on Android mobile devices.

Output: Inference predictions, per-image execution time, power consumption logs, and derived energy efficiency metrics.

Scoring recipe

def compute_metrics(predictions, ground_truth, runtimes_ms, power_mW):
    accuracy = sum(p == g for p, g in zip(predictions, ground_truth)) / len(ground_truth) * 100
    avg_runtime = sum(runtimes_ms) / len(runtimes_ms)
    fps = 1000 / avg_runtime
    energy_efficiency = fps / (power_mW / 1000)
    return accuracy, avg_runtime, energy_efficiency

Common pitfalls

• Power profiling is only reported for Snapdragon 820, not Snapdragon 855.
• TFLite GPU execution crashes on large models (VGG16), preventing direct runtime comparison.
• Accuracy degradation is inherent to binarization and varies significantly across architectures.

Evidence (verbatim from paper)

The metrics of accuracy, model size, runtime, power and energy consumption are evaluated and compared to demonstrate the efficiency of PhoneBit. ... PhoneBit consumes around 226mW power and obtains 105.26 FPS per watt.

Citation

@misc{chen2019phonebit,
  title={PhoneBit: Efficient GPU-Accelerated Binary Neural Network Inference Engine for Mobile Phones},
  author={Chen et al. (2019)},
  year={2019},
  note={arXiv:1912.04050}
}

• arXiv: 1912.04050