Cv Inference Eval

cv-inference-eval

eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations — Bamberg et al. (2025) (arXiv:2509.14388, 2025)

What this evaluates

Evaluates end-to-end inference latency and hardware efficiency of computer vision models on edge AI hardware. It probes how well a hardware-software co-design optimizes data movement and compute utilization under strict memory and bandwidth constraints.

Datasets

• ImageNet — total ?; splits: test (-1)
• COCO 2017 — total ?; splits: test (-1)

Metrics

• Latency [ms] (primary) — range: other
  • Batch-size-1 end-to-end inference time measured on silicon. Lower values indicate faster execution.
• LTP — range: other
  • Latency-TOPS Product, defined as latency multiplied by hardware TOPS. Lower values indicate higher efficiency, meaning less silicon is required to achieve a given performance.

Input / output format

Input: INT8-quantized computer vision model (classification, detection, or segmentation) exported to LiteRT v2.18.0, executed on a production SoC with 2 TOPS NPU, 12 GB/s DDR bandwidth, and 1 MB SRAM.

Output: End-to-end inference latency in milliseconds and the computed Latency-TOPS Product (LTP) metric.

Scoring recipe

def evaluate(model, hardware_top_s):
    # Measure batch-size-1 end-to-end inference time on silicon
    latency_ms = measure_batch1_inference_time(model, hardware_config)
    # Compute efficiency metric
    ltp = latency_ms * hardware_top_s
    return {"latency_ms": latency_ms, "ltp": ltp}

Common pitfalls

• iNPU latency is approximated as the inverse of reported throughput, which is a theoretical lower bound that may overestimate actual performance due to pipeline depth.
• Models are strictly INT8 quantized; accuracy is not reported, so comparisons rely solely on latency and efficiency metrics.
• Comparing systems with different TOPS and bandwidth requires using LTP rather than raw latency to normalize for hardware capability.

Evidence (verbatim from paper)

Real-life (silicon) performance is measured as batch-size-1 end-to-end inference latency on a production, mass-market micro-processor-unit SoC (MPU (micro-processor-unit SoC))–embedding the proposed 2-TOPS NPU with 1MiB of SRAM and 12GB/s of DDR bandwidth—employing LiteRT benchmarking tools. To capture performance efficiency—and thereby power and area—we introduce a second metric in addition to latency: the Latency–TOPS Product (LTP), defined as: LTP ≜ latency·TOPS, where lower values indicate higher efficiency, meaning less silicon hardware is required to achieve a given performance.

Citation

@misc{bamberg2025eiqneutron,
  title={eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations},
  author={Bamberg et al. (2025)},
  year={2025},
  note={arXiv:2509.14388}
}

• arXiv: 2509.14388