Kws Accuracy Eval

kws-accuracy-eval

Temporal Convolution for Real-time Keyword Spotting on Mobile Devices — Choi et al. (2019) (arXiv:1904.03814, 2019)

What this evaluates

This benchmark evaluates keyword spotting models on mobile devices by measuring classification accuracy, computational cost (FLOPs, parameters), and real-time inference latency on one-second audio utterances. It specifically tests whether temporal convolutions can replace 2D convolutions to reduce computational load while maintaining or improving accuracy.

Datasets

• Google Speech Commands Dataset — total 64727; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified utterances out of the total test set instances, averaged over 15 independent training runs.
• FLOPs — range: other
  • Total floating-point operations computed by the model, measured using the TensorFlow profiling tool.
• number of parameters — range: other
  • Total count of all model parameters (including non-trainable), extracted directly from the model architecture.
• inference time — range: other
  • Average time to process one utterance on a Google Pixel 1 (single little core), measured 50 times per model using the TensorFlow Lite Android benchmark tool.

Input / output format

Input: One-second audio utterance files, preprocessed into a sequence of 40 MFCC features per frame (30 ms window, 10 ms stride), stacked over the time axis.

Output: Class label prediction from 12 classes: 'yes', 'no', 'up', 'down', 'left', 'right', 'on', 'off', 'stop', 'go', 'silence', and 'unknown'.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• FLOPs do not always correlate with actual inference speed on mobile hardware; real-world latency must be measured on target devices.
• Parameter counts must include all parameters, not just trainable ones, as prior works often reported only trainable parameters.
• Inference time measurement must start from the first layer receiving MFCC inputs to isolate model performance from preprocessing overhead.

Evidence (verbatim from paper)

We use accuracy as the main metric to evaluate how well the model performs. We trained each model 15 times and report its average performance. We report the number of operations and parameters which faithfully reflect the real-world environment for mobile deployment. Unlike previous works which only reported the numbers for part of the computation such as the number of multiply operations or the number of multiplications and additions only in the matrix-multiplication operations, we include FLOPs, computed by TensorFlow profiling tool, and the number of all parameters instead of only trainable parameters reported by previous studies. Therefore, we also measure inference time on a mobile device using the TensorFlow Lite Android benchmark tool. The benchmark program measures the inference time 50 times for each model and reports the average.

Citation

@misc{choi2019temporal,
  title={Temporal Convolution for Real-time Keyword Spotting on Mobile Devices},
  author={Choi et al. (2019)},
  year={2019},
  note={arXiv:1904.03814}
}

• arXiv: 1904.03814