qhjqhj00 imagenet-mcu-eval
Tiny Machine Learning: Progress and Futures — Ji Lin et al. (2024) (arXiv:2403.19076, 2024)
What this evaluates
Evaluates image classification accuracy on ultra-constrained microcontrollers (MCUs) with strict SRAM and Flash limits, measuring the trade-off between model quantization, memory footprint, and performance.
Datasets
- ImageNet — total ?; splits: val (-1)
Metrics
Top-1 accuracy(primary) — range: percent- Percentage of validation images where the model's highest-confidence prediction matches the ground truth label.
Top-5 accuracy— range: percent- Percentage of validation images where the ground truth label appears in the model's top 5 predictions.
Input / output format
Input: Standard ImageNet validation images processed through a quantized (int8/int4) neural network optimized for MCU hardware constraints.
Output: Predicted class labels (top-1 and top-5) for each input image.
Scoring recipe
For each image in the ImageNet validation set:
pred = model.predict(image)
if pred == ground_truth: top1_correct += 1
if ground_truth in pred[:5]: top5_correct += 1
top1_acc = (top1_correct / total_images) * 100
top5_acc = (top5_correct / total_images) * 100
Common pitfalls
- Hardware memory limits (SRAM/Flash) are hard constraints; exceeding them causes out-of-memory (OOM) failures rather than just slowdowns.
- Quantization precision (int8 vs int4) significantly impacts both accuracy and memory footprint, requiring careful system-algorithm co-design.
Evidence (verbatim from paper)
We compared MCUNet with existing state-of-the-art solutions on ImageNet classification under two hardware settings: 256kB SRAM/1MB Flash and 512kB SRAM/2MB Flash. The goal is to achieve the highest ImageNet Top-1 accuracy on resource-constrained MCUs (Table IV).
Citation
@misc{lin2024tinyml,
title={Tiny Machine Learning: Progress and Futures},
author={Ji Lin et al. (2024)},
year={2024},
note={arXiv:2403.19076}
}
- arXiv: 2403.19076