Ml Accelerator Inference Eval

ml-accelerator-inference-eval

Survey and Benchmarking of Machine Learning Accelerators — Reuther et al. (2019) (arXiv:1908.11348, 2019)

What this evaluates

Evaluates the inference latency, power consumption, and computational throughput of various machine learning accelerators and CPUs on object detection tasks. It probes the real-world SWaP (Size, Weight, and Power) efficiency and performance discrepancies between advertised and actual hardware capabilities.

Datasets

• Microsoft COCO — total ?; splits: test (-1)

Metrics

• Avg. Single Image Inference Time (ms) (primary) — range: other
  • Average time in milliseconds to process a single image through the neural network.
• Measured GOPS/W — range: other
  • Measured computational throughput in giga operations per second divided by measured power consumption in watts.
• Avg. Model Load Time (s) — range: other
  • Average time in seconds to load the entire neural network model onto the device for each image.
• Measured Power (W) — range: other
  • Average power draw in watts measured via USB multimeter during inference.

Input / output format

Input: Single images from the Microsoft COCO dataset, processed sequentially with batch size = 1.

Output: Inference predictions (object detection outputs), though the benchmark primarily records latency and power metrics rather than prediction accuracy.

Scoring recipe

for each device:
  load_model(device)
  total_time = 0
  total_power = 0
  for image in dataset:
    start = time.now()
    predict = run_inference(device, image)
    end = time.now()
    total_time += (end - start)
    total_power += measure_power_via_multimeter(device)
  avg_inference_ms = (total_time / N) * 1000
  avg_power_w = total_power / N
  gops_w = measured_gops / avg_power_w

Common pitfalls

• Software throttles enforce batch size = 1 and require full model reload per image, which underestimates production performance.
• Power is measured via USB multimeter rather than system-level power draw, potentially missing overhead from host CPU/memory.
• Reported hardware specs (GOPS, GOPS/W) often significantly overstate actual measured performance, especially for NCS2.

Evidence (verbatim from paper)

Table I summarizes the reported and measured giga operations per second (GOPS), power (W), and GOPS/W along with average model load time in seconds and average single image inference time in milliseconds.

Citation

@misc{reuther2019survey,
  title={Survey and Benchmarking of Machine Learning Accelerators},
  author={Reuther et al. (2019)},
  year={2019},
  note={arXiv:1908.11348}
}

• arXiv: 1908.11348