Hw Nas Bench Eval

hw-nas-bench-eval

HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark — Li et al. (2021) (arXiv:2103.10584, 2021)

What this evaluates

Evaluates hardware-aware neural architecture search (HW-NAS) algorithms by measuring how effectively they discover network topologies that optimize the trade-off between classification accuracy and on-device inference latency for specific target hardware.

Datasets

• HW-NAS-Bench — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RICE-EIC/HW-NAS-Bench

Metrics

• top-1 accuracy (primary) — range: percent
  • Percentage of correctly classified samples in the test set.
• latency (ms) — range: other
  • On-device inference time measured in milliseconds for a single forward pass.

Input / output format

Input: NAS search space definition (e.g., FBNet), target dataset (e.g., CIFAR-100), and target hardware device specification.

Output: A searched neural network architecture (topology and weights) optimized for the specified target device.

Scoring recipe

def evaluate(architecture, dataset, target_device, hw_bench):
    acc = compute_top1_accuracy(architecture, dataset)
    lat = hw_bench.query_latency(architecture, target_device)
    return {"top-1 accuracy": acc, "latency (ms)": lat}

Common pitfalls

• Assuming latency or energy measurements from one hardware device transfer directly to another without re-measurement.
• Optimizing for a generic latency constraint rather than querying the exact device-specific hardware cost provided by the benchmark.
• Ignoring the negligible overhead of querying the benchmark during the search process, which can skew runtime comparisons.

Evidence (verbatim from paper)

Benchmark Setting. We adopt a SOTA HW-NAS algorithm, ProxylessNAS*(Cai et al., [2018])* for this experiment. As an example to use our HW-NAS-Bench, we use ProxylessNAS to search over the FBNet*(Wu et al., [2019])* search space on CIFAR-100*(Krizhevsky et al., [2009])*, when targeting different devices in our HW-NAS-Bench by simply querying the corresponding device’s measured/estimated hardware-cost, which has negligible overhead as compared to the HW-NAS algorithm itself, without the need for hardware expertise or knowledge during the whole HW-NAS. Table 5 illustrates that the searched architectures achieve the lowest latency among all architectures when the target devices of HW-NAS are the same as the one used to measure the architecture’s on-device inference latency.

Citation

@misc{li2021hwnasbench,
  title={HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2103.10584}
}

• arXiv: 2103.10584