Keyword Spotting Efficiency Eval

keyword-spotting-efficiency-eval

Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware — Blouw et al. (2018) (arXiv:1812.01739, 2018)

What this evaluates

Evaluates the energy efficiency and inference speed of various hardware platforms (CPU, GPU, neuromorphic chips) running a keyword spotting neural network on audio data. It measures how power consumption and latency scale with network size and batch configuration.

Datasets

• Keyword Spotting Dataset — total ?; splits: test (-1); repo https://github.com/abr/power_benchmarks

Metrics

• energy cost per inference (J) (primary) — range: Joules (J)
  • Dynamic power (W) = mean runtime power (W) - mean idle power (W). Energy cost per inference (J) = dynamic power (W) / inference speed (inferences/sec).

Input / output format

Input: Audio frames with a 10ms stride fed into a two-layer spiking neural network.

Output: Keyword classification output, plus logged runtime power, idle power, and inference timestamps for metric computation.

Scoring recipe

dynamic_power_watts = mean_runtime_power_watts - mean_idle_power_watts
energy_cost_joules = dynamic_power_watts / inferences_per_second
# Compute 95% CI via bootstrapping on log-transformed values
ci_lower, ci_upper = bootstrap_ci(log(energy_cost_joules), n=1000, alpha=0.05)

Common pitfalls

• Loihi and Movidius only support a batch size of 1, making direct batched comparisons invalid.
• Scaling experiments fix batch size at 1 and ignore trained weights, measuring only topology/size effects on speed and energy.
• Confidence intervals must be computed on log-transformed power values, not raw values.

Evidence (verbatim from paper)

To explain, the mean idle power consumption is subtracted from the mean runtime power consumption to give the mean dynamic power consumption in Watts in the fourth column. Since one Watt is equivalent to one joule per second, we can divide the dynamic power consumption value by the number of inferences per second, listed in the fifth column, to get the mean energy cost per inference, listed in the final column.

Citation

@misc{blouw2018benchmarking,
  title={Benchmarking Keyword Spotting Efficiency on Neuromorphic Hardware},
  author={Blouw et al. (2018)},
  year={2018},
  note={arXiv:1812.01739}
}

• arXiv: 1812.01739