Mlperf Tiny Nas Eval

mlperf-tiny-nas-eval

Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks — Risso et al. (2022) (arXiv:2206.00302, 2022)

What this evaluates

Evaluates whether differentiable NAS methods can discover neural architectures that maximize classification accuracy while minimizing energy consumption and latency. The protocol enforces strict weight memory constraints on edge hardware and measures real-world deployment metrics on the NUCLEO-H743ZI2 MCU.

Datasets

• Image Classification (CIFAR-10) — total 60000; splits: test (-1)
• Visual Wake Word (MSCOCO 2014) — total 109619; splits: test (-1)
• KeyWord Spotting (Speech Commands v2) — total 105829; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Test-set classification accuracy calculated as the fraction of correctly predicted labels over the total number of test instances.
• energy_consumption — range: other
  • Average energy per inference measured in millijoules (mJ) on the target NUCLEO-H743ZI2 MCU after converting the model to C code.
• latency — range: other
  • Average inference time in milliseconds (ms) measured on the target NUCLEO-H743ZI2 MCU.
• memory_footprint — range: other
  • Weight memory size in kilobytes (kB) reported as a percentage difference from the imposed size constraint.
• OPs — range: other
  • Total number of floating-point operations required for inference, used as a proxy for computational complexity.

Input / output format

Input: 32×32×3 RGB images (IC), 96×96×3 RGB images (VWW), or audio utterances (KWS).

Output: Class labels (10 classes for IC, binary person-presence for VWW, 12 classes for KWS).

Scoring recipe

def compute_metrics(predictions, gold_labels, model):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    # Hardware deployment metrics (NUCLEO-H743ZI2 via X-Cube-AI)
    energy_mj = measure_energy_on_hardware(model)
    latency_ms = measure_latency_on_hardware(model)
    memory_kb = model.weights_size_kb
    ops = model.total_flops
    return accuracy, energy_mj, latency_ms, memory_kb, ops

Common pitfalls

• Memory constraint strictly applies to weight memory (kB), not total model size including activations or code.
• Energy and latency are hardware-measured on the NUCLEO-H743ZI2 MCU, not estimated from FLOPs or simulators.
• The Anomaly Detection benchmark was explicitly excluded because its autoencoder architecture lacks degrees of freedom to trade OPs for size.

Evidence (verbatim from paper)

Table I summarizes the deployment results on the NUCLEO-H743ZI2 for the IC and VWW benchmarks. The Mem. column reports the memory occupation of each model, and the difference in percentage from the imposed constraint. As shown, all networks are within ± 3.3% from the target, showing that our constraint formulation produces the expected results. Further, on the IC task we find solutions with energy consumption spanning from 25.7mJ to 13.4mJ respectively for the 75%, 50% and 25% targets. Noteworthy, the 75%-H network reduces the energy consumption by 2.2× with respect to the seed, whit negligible accuracy drop.

Citation

@misc{risso2022multicomplexitylossdnas,
  title={Multi-Complexity-Loss DNAS for Energy-Efficient and Memory-Constrained Deep Neural Networks},
  author={Risso et al. (2022)},
  year={2022},
  note={arXiv:2206.00302}
}

• arXiv: 2206.00302