Mnn Inference Benchmark Eval

mnn-inference-benchmark-eval

MNN: A Universal and Efficient Inference Engine — Jiang et al. (2020) (arXiv:2002.12418, 2020)

What this evaluates

Evaluates the inference latency and computational efficiency of mobile deep learning engines across different hardware platforms, compute backends, and neural network architectures.

Datasets

• MobileNet-v1 — total ?; splits: test (-1)
• SqueezeNet-v1.1 — total ?; splits: test (-1)
• ResNet-18 — total ?; splits: test (-1)
• Inception-v3 — total ?; splits: test (-1)

Metrics

• inference time (ms) (primary) — range: other
  • Average latency in milliseconds to process a single 224×224 RGB image (batch size 1) over 10 runs, preceded by one warm-up inference.

Input / output format

Input: Single 224×224 RGB image

Output: Model predictions (evaluation measures execution latency rather than output format or correctness)

Scoring recipe

warmup_run(model, input_image)
latencies = []
for _ in range(10):
    start = time()
    model(input_image)
    latencies.append(time() - start)
return sum(latencies) / len(latencies)

Common pitfalls

• Omitting the mandatory warm-up inference run before benchmarking
• Failing to average over exactly 10 runs or using a different batch size/input resolution
• Comparing GPU backends without specifying the underlying API (Metal, OpenCL, OpenGL, or Vulkan)

Evidence (verbatim from paper)

We report the inference time of one 224 × 224 RGB image (i.e., batch size is 1), averaged by 10 runs. Before benchmark, one warm-up inference is conducted for fair comparison with other works (Tencent, 2017; Google, 2017a).

Citation

@misc{jiang2020mnn,
  title={MNN: A Universal and Efficient Inference Engine},
  author={Jiang et al. (2020)},
  year={2020},
  note={arXiv:2002.12418}
}

• arXiv: 2002.12418