Proai Hardware Benchmark Eval

proai-hardware-benchmark-eval

ProAI: An Efficient Embedded AI Hardware for Automotive Applications -- a Benchmark Study — Mantowsky et al. (2021) (arXiv:2108.05170, 2021)

What this evaluates

Evaluates the power efficiency, throughput, and real-time inference performance of embedded AI hardware platforms running multitask and single-task deep neural networks for automotive vision tasks.

Datasets

• COCO test2017 — total 41000; splits: test (5000)

Metrics

• FPS, inference time, memory usage, energy efficiency (Wtotal, W/fps) (primary) — range: other
  • FPS = total_images / total_inference_time. Inference time = total_inference_time / total_images. Memory usage = peak logged RAM/VRAM. Energy efficiency = Wtotal (total power) and W/fps (power per frame).

Input / output format

Input: RGB images from the COCO test2017 dataset (5,000 images), passed through standard pre-processing pipelines for object detection, semantic segmentation, and human pose estimation.

Output: Model predictions (bounding boxes, segmentation masks, pose keypoints) and system telemetry logs (CPU/GPU utilization, memory, temperature, power, latency) recorded at 0.5 Hz.

Scoring recipe

def compute_metrics(logs, num_images=5000):
    total_time = logs['inference_time']
    fps = num_images / total_time
    inference_time = total_time / num_images
    memory_usage = max(logs['memory'])
    w_total = max(logs['power'])
    w_per_fps = w_total / fps
    return {'FPS': fps, 'inference_time': inference_time, 'memory_usage': memory_usage, 'Wtotal': w_total, 'W/fps': w_per_fps}

Common pitfalls

• Comparing pre/post-processing times across different CPU architectures (ARM vs x86) without accounting for architectural differences.
• Using the full 41k COCO test set instead of the specified 5k subset, which alters throughput and memory profiling results.
• Measuring training metrics instead of inference-only metrics, as the benchmark explicitly focuses on deployment and real-time inference.

Evidence (verbatim from paper)

The most essential part for mobile applications is efficiency and the amount of throughput per second of the hardware. Therefore, the metrics frames-per-second (FPS), inference time, memory usage and energy efficiency (W) (Wtotal and W/fps) were used for the benchmark. For the benchmark evaluation, we use the specifications given in Table 1 as the maximum GPU power consumption.

Citation

@misc{mantowsky2021proai,
  title={ProAI: An Efficient Embedded AI Hardware for Automotive Applications -- a Benchmark Study},
  author={Mantowsky et al. (2021)},
  year={2021},
  note={arXiv:2108.05170}
}

• arXiv: 2108.05170