Mobile Gpu Inference Eval

mobile-gpu-inference-eval

HG-Caffe: Mobile and Embedded Neural Network GPU (OpenCL) Inference Engine with FP16 Supporting — Ji (2019) (arXiv:1901.00858, 2019)

What this evaluates

This benchmark evaluates the inference performance of a mobile GPU-accelerated neural network engine. It measures execution latency and peak memory consumption across different hardware platforms, precision formats (FP32 vs FP16), and batch sizes. It probes the engine's ability to leverage GPU parallelism and half-precision arithmetic for efficient edge deployment.

Datasets

• MNIST — total ?; splits: test (-1)
• Cifar-10 — total ?; splits: test (-1)
• Style Transfer Dataset — total ?; splits: test (-1)

Metrics

• execution time (primary) — range: other
  • Wall-clock time in seconds to process a single forward pass for a given batch size.
• peak memory usage — range: other
  • Maximum RAM or GPU memory allocated during the inference run, measured in gigabytes.

Input / output format

Input: Batched input tensors for convolutional neural networks (LeNet, AlexNet, ResNet-20, Style Transfer Net) with batch sizes ranging from 1 to 1024.

Output: Inference outputs (logits/features) and system performance counters: execution time (s) and peak memory usage (GB).

Scoring recipe

def score(predictions, gold, config):
    time_s = config['measured_time_s']
    mem_gb = config['peak_memory_gb']
    throughput = 1.0 / time_s if time_s > 0 else 0.0
    return {'time_s': time_s, 'memory_gb': mem_gb, 'throughput': throughput}

Common pitfalls

• GPU speedup is highly dependent on batch size; small batches (e.g., 1) may show CPU superiority due to kernel launch overhead.
• Hardware-specific memory limits (e.g., Kirin 970's ~100MB GPU kernel allocation) can cause page thrashing, making raw speed comparisons misleading without accounting for memory constraints.
• The evaluation only measures latency and memory, not model accuracy or precision loss from FP16 conversion.

Evidence (verbatim from paper)

We compared the execution time and peak memory usage of CPU with single precision, GPU with single precision and GPU with half precision for each neural network.

Citation

@misc{ji2019hgcaffe,
  title={HG-Caffe: Mobile and Embedded Neural Network GPU (OpenCL) Inference Engine with FP16 Supporting},
  author={Ji (2019)},
  year={2019},
  note={arXiv:1901.00858}
}

• arXiv: 1901.00858