Tinymlperf Eval

tinymlperf-eval

MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers — Banbury et al. (2020) (arXiv:2010.11267, 2020)

What this evaluates

Evaluates the deployment efficiency and accuracy of neural networks on commodity microcontrollers (MCUs) under strict memory and latency constraints. It measures inference speed, memory footprint, and task-specific accuracy across vision, audio, and anomaly detection workloads.

Datasets

• TinyMLPerf (VWW, KWS, AD) — total ?; splits: test (-1); repo https://github.com/ARM-software/ML-zoo

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly classified instances on the held-out test set for VWW and KWS tasks.
• Latency (ms) (primary) — range: other
  • End-to-end inference time measured on the target MCU using the Mbed Timer API.
• AUC (%) — range: percent
  • Area Under the Receiver Operating Characteristic Curve for the Anomaly Detection task.
• Uptime (%) — range: percent
  • Ratio of model inference latency to the input stride time, representing the MCU duty cycle.
• SRAM / Flash Usage (KB) — range: other
  • Peak activation memory (SRAM) and model binary size (Flash) measured via TFLM recording APIs and Mbed compiler.

Input / output format

Input: Preprocessed input tensors (images, audio spectrograms, or time-series features) passed to a TFLite model via the TensorFlow Lite for Microcontrollers (TFLM) runtime.

Output: Class labels or anomaly scores, alongside hardware metrics (latency, SRAM, Flash) captured during runtime execution.

Scoring recipe

def evaluate(predictions, gold, latency_ms, sram_kb, flash_kb, stride_ms, task):
    if task in ['VWW', 'KWS']:
        acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold) * 100
        return {'Accuracy (%)': acc, 'Latency (ms)': latency_ms, 'SRAM (KB)': sram_kb, 'Flash (KB)': flash_kb}
    elif task == 'AD':
        auc = compute_auc(gold, predictions) * 100
        uptime = (latency_ms / stride_ms) * 100
        return {'AUC (%)': auc, 'Uptime (%)': uptime, 'SRAM (KB)': sram_kb, 'Flash (KB)': flash_kb}

Common pitfalls

• Latency and memory measurements include TFLM runtime overheads, not just model weights.
• Uptime metric depends heavily on the input stride time, making cross-task comparisons difficult without normalization.
• Sub-byte quantization (e.g., 4-bit) requires software emulation on Cortex-M, which can hide latency benefits if instruction-level parallelism isn't fully exploited.
• Flash and SRAM constraints vary by MCU target (small/medium/large), so models are not directly comparable across hardware tiers.

Evidence (verbatim from paper)

We measure latency on the MCU using the Mbed Timer API. The eFlash occupancy is determined using the Mbed compiler and the SRAM consumption is obtained using the TFLM recording memory APIs. We trained all the models with exactly the same training recipe and quantized them to 8-bit weights and activations (including input) before measuring their accuracy.

Citation

@misc{banbury2020micronets,
  title={MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers},
  author={Banbury et al. (2020)},
  year={2020},
  note={arXiv:2010.11267}
}

• arXiv: 2010.11267