Duccio Nas Eval

duccio-nas-eval

Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices — Burrello et al. (2023) (arXiv:2310.07217, 2023)

What this evaluates

Evaluates hardware-aware neural architecture search (NAS) methods on edge IoT tasks, measuring classification accuracy alongside hardware constraints like memory footprint, latency, and computational complexity (OPs) on a RISC-V IoT SoC. It benchmarks both mask-based and path-based differentiable NAS approaches across image classification, visual wake words, keyword spotting, and anomaly detection tasks.

Datasets

• CIFAR-10 — total 60000; splits: test (-1)
• MSCOCO 2014 — total 100000; splits: test (-1)
• Speech Commands v2 — total 105829; splits: test (-1)
• DCASE2020 — total ?; splits: test (-1)
• Tiny ImageNet — total 100000; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard top-1 classification accuracy: fraction of correctly predicted labels over total test samples.
• Memory footprint — range: other
  • Total model parameter and activation memory size in kilobytes (kB) on the target hardware.
• Latency — range: other
  • Inference time in milliseconds (ms) measured on the GAP8 SoC after 8-bit quantization and DORY compilation.
• OPs — range: other
  • Million Operations (MOPs) representing computational complexity of the network.

Input / output format

Input: RGB images (32x32, 96x96, or 64x64) or audio utterances.

Output: Class predictions (or anomaly scores) and hardware metrics (memory in kB, latency in ms, OPs in MOPs) measured on GAP8 SoC.

Scoring recipe

def compute_metrics(predictions, gold, model, hardware):
    accuracy = (predictions == gold).sum() / len(gold)
    memory_kb = model.get_memory_footprint()
    latency_ms = hardware.measure_inference_time(model)
    ops_mops = model.count_operations() / 1e6
    return accuracy, memory_kb, latency_ms, ops_mops

Common pitfalls

• Using the reduced MLPerf Tiny test set for CIFAR-10 instead of the full set, which yields less stable accuracy results.
• Evaluating latency without accounting for the GAP8 memory hierarchy (L1/L2/L3) and DMA co-processors, which significantly impacts real-world inference time.
• Forgetting to quantize models to 8-bit before deployment on GAP8, as the SoC lacks an FPU.

Evidence (verbatim from paper)

All results are reported on test sets. ... The Mem. column reports the memory occupation of each model. ... Regarding latency, we find solutions that span from 9.40ms to 21.26ms and 12.59ms to 27.86ms, respectively, for the mask-based DNAS and the path-based DNAS.

Citation

@misc{burrello2023duccio,
  title={Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices},
  author={Burrello et al. (2023)},
  year={2023},
  note={arXiv:2310.07217}
}

• arXiv: 2310.07217