Edge AI Platform Inference Eval

edge-ai-platform-inference-eval

Benchmarking Edge AI Platforms for High-Performance ML Inference — Jayanth et al. (2024) (arXiv:2409.14803, 2024)

What this evaluates

Evaluates the inference performance of heterogeneous edge AI platforms (CPU, GPU, NPU) across fundamental linear algebra primitives and diverse neural network models. It probes hardware efficiency in compute-bound versus memory-bound workloads, batch processing scalability, and quantization support.

Datasets

• Matrix Multiplication — total ?; splits: (unstated)
• Matrix-Vector Multiplication — total ?; splits: (unstated)
• Dot Product — total ?; splits: (unstated)
• MobileNetV2 — total ?; splits: (unstated)
• LSTM — total ?; splits: (unstated)
• TinyLlama — total ?; splits: (unstated)

Metrics

• Latency (ms) (primary) — range: other
  • Total time taken to generate output from when an input is provided, measured in milliseconds.
• Avg Throughput (GFLOPS) — range: other
  • Average number of floating-point operations performed per second, calculated as total FLOPs divided by execution time, measured in GigaFLOPS.
• FPS — range: other
  • Frames Per Second; the number of individual images processed per second, used specifically for video classification throughput.

Input / output format

Input: Linear algebra: square matrices A (N×N) and B (N×N), or vectors x (N×1) and y (N×1) with N ranging from 16 to 1024. Neural networks: video frames (MobileNetV2), time-series sequences (LSTM), and text tokens (TinyLlama). Batch sizes are varied for video classification tasks.

Output: Standard model predictions/classifications and hardware performance measurements (latency in ms, throughput in GFLOPS, or FPS). No custom output formatting is required beyond standard inference results.

Scoring recipe

def compute_metrics(execution_times_ms, total_flops, batch_size, task_type):
    avg_latency_ms = sum(execution_times_ms) / len(execution_times_ms)
    avg_throughput_gflops = (total_flops / (avg_latency_ms / 1000.0)) / 1e9
    fps = None
    if task_type == 'video_classification':
        fps = batch_size / (avg_latency_ms / 1000.0)
    return {'latency_ms': avg_latency_ms, 'throughput_gflops': avg_throughput_gflops, 'fps': fps}

Common pitfalls

• Precision mismatch: The paper exclusively uses FP-16 for uniform analysis across platforms, despite CPUs/GPUs natively supporting FP-32.
• Framework overhead: OpenVINO and IPEX use lazy evaluation and asynchronous execution, which can skew latency measurements if warm-up runs are omitted or if results are not properly synchronized.
• Batch size sensitivity: Performance characteristics shift dramatically between compute-bound (high batch) and memory-bound (low batch) regimes, requiring explicit reporting of batch sizes for reproducibility.

Evidence (verbatim from paper)

The metrics we use for these operations are latency and average throughput. For LSTM and LLM models, inference latency will be the performance metric, and for video classification, along with inference latency, FPS (Frames Per Second) will be used as a measure of throughput.

Citation

@misc{jayanth2024edgeai,
  title={Benchmarking Edge AI Platforms for High-Performance ML Inference},
  author={Jayanth et al. (2024)},
  year={2024},
  note={arXiv:2409.14803}
}

• arXiv: 2409.14803