Match Compiler Eval

match-compiler-eval

MATCH: Model-Aware TVM-based Compilation for Heterogeneous Edge Devices — Hamdi et al. (2024) (arXiv:2410.08855, 2024)

What this evaluates

Evaluates a model-aware compiler framework for deploying deep neural networks on heterogeneous edge microcontrollers. It measures execution latency, hardware utilization efficiency (MACs/cycle), and scheduling robustness under memory constraints across multiple standard DNN architectures.

Datasets

• MLPerf Tiny Benchmark Suite — total ?; splits: test (-1)

Metrics

• Latency (ms) (primary) — range: other
  • End-to-end execution time measured in milliseconds using on-board dedicated performance counters on the target hardware.
• Speed-up — range: other
  • Ratio of baseline compiler latency to MATCH compiler latency (baseline_latency / match_latency).
• MACs/cycle — range: other
  • Number of multiply-accumulate operations executed per clock cycle, indicating hardware utilization efficiency.

Input / output format

Input: Deep neural network models (ResNet V1, MobileNetV1, DS-CNN, FC Autoencoder) and custom convolutional blocks with specified hyperparameters (spatial dimensions, channels, kernel size, stride, dilation, padding).

Output: Execution latency in milliseconds, speed-up ratios relative to baselines, and MACs/cycle efficiency metrics.

Scoring recipe

latency_ms = measure_on_hardware(model, target_hw, freq_mhz=260)
speedup = baseline_latency_ms / latency_ms
macs_per_cycle = total_macs / (latency_ms * freq_mhz * 1e-3)
return latency_ms, speedup, macs_per_cycle

Common pitfalls

• NNTool uses highly optimized, hyperparameter-specific kernels, making direct latency comparisons unfair to MATCH's general-purpose template.
• L1 memory size drastically impacts scheduling; results at constrained memory (e.g., 8-16 kB) differ significantly from full-memory baselines.
• Cost model discrepancies on specific accelerators (e.g., NE16) can cause suboptimal layer offloading decisions, affecting reported latency.

Evidence (verbatim from paper)

Latency measurements were obtained using on-board dedicated performance counters. Our tool is compared with TVM (which only generates code for the main MCU of the two systems) and with HTVM (for DIANA) and DORY (GAP9), which utilize the same platform backend libraries and HW modules but different AI compilation steps. Additionally, on GAP9, we also compare MATCH with the NNTool, a proprietary tool from GreenWaves that utilizes a different backend library. Noteworthy, all the following results are measured on the actual HW platforms.

Citation

@misc{hamdi2024match,
  title={MATCH: Model-Aware TVM-based Compilation for Heterogeneous Edge Devices},
  author={Hamdi et al. (2024)},
  year={2024},
  note={arXiv:2410.08855}
}

• arXiv: 2410.08855