Mlperf Edge Inference Eval

mlperf-edge-inference-eval

Performance Characterization of using Quantization for DNN Inference on Edge Devices: Extended Version — Hyunho Ahn et al. (arXiv:2303.05016, 2023)

What this evaluates

This protocol evaluates the inference performance of DNN models under various quantization precisions (FP16/INT8, static/dynamic) across multiple inference frameworks on edge and server hardware. It measures standard accuracy and latency metrics using the MLPerf Edge Inference benchmark suite.

Datasets

• ImageNet ILSVRC2012 — total ?; splits: val (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Top-1 classification accuracy on the ImageNet validation set, computed by comparing model predictions against ground-truth labels.
• latency — range: ms
  • Average time per inference request measured by MLPerf LoadGen across offline and online scenarios.

Input / output format

Input: Input images resized per model specification, rescaled to [0.0, 1.0], and normalized using mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225].

Output: Inference predictions and performance logs (latency, throughput, accuracy) generated by MLPerf LoadGen.

Scoring recipe

# MLPerf LoadGen handles metric computation
predictions = model.run(input_images)
accuracy = sum(pred == gold for pred, gold in zip(predictions, labels)) / len(labels)
latency = mean(inference_times)
throughput = num_requests / total_time

Common pitfalls

• Hardware-specific instruction sets (e.g., VNNI on Intel) and backend optimizations can cause framework performance to diverge significantly from theoretical quantization benefits.
• Static quantization may cause accuracy drops not seen in dynamic quantization, requiring careful calibration dataset selection.

Evidence (verbatim from paper)

MLPerf Edge Inference benchmark suite v2.1 has been used in this study. This suite contains the LoadGen python module—responsible for generating input traffic—-that is built with the default setting. ... The validation dataset of ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) is used to input data for all models under test. Input images are resized for the size that is suggested on each model. Final values are rescaled to $[0.0,1.0]$ and then normalized using the mean value of $[0.485,0.456,0.406]$ and standard deviation of $[0.229,0.224,0.225]$.

Citation

@misc{ahn2023mlperfquantization,
  title={Performance Characterization of using Quantization for DNN Inference on Edge Devices: Extended Version},
  author={Hyunho Ahn et al.},
  year={2023},
  note={arXiv:2303.05016}
}

• arXiv: 2303.05016