Speech Commands Eval

speech-commands-eval

Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition — Pete Warden (arXiv:1804.03209, 2018)

What this evaluates

This benchmark evaluates limited-vocabulary keyword spotting models for on-device speech recognition. It probes a model's ability to correctly identify isolated spoken words from a fixed set of 10 commands, while also handling background silence and unrecognized speech in both aligned and continuous streaming audio contexts.

Datasets

• Speech Commands — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Top-One Error (primary) — range: percent
  • Calculated as the proportion of audio clips where the model's top predicted class matches the ground truth label. Evaluated on a fixed test set of 12 categories (10 target words, 'Unknown Word', and 'Silence') with equal representation per class.
• Streaming Error Metrics — range: percent
  • Measures performance on continuous audio streams. Includes Matched % (correctly identified within time tolerance), Correct % (distinguished from noise but wrong label), and False Positive % (detected in silence). Default time tolerance is 750ms.

Input / output format

Input: One-second isolated WAV audio files, optionally arranged in subfolders by label for batch evaluation, or continuous multi-minute WAV streams for streaming tests.

Output: A single predicted class label per audio clip (e.g., 'Yes', 'No', 'Silence', 'Unknown Word') or a sequence of timestamped predictions for streaming evaluation.

Scoring recipe

def compute_top_one_error(predictions, ground_truth):
    correct = sum(1 for p, g in zip(predictions, ground_truth) if p == g)
    return (1 - correct / len(ground_truth)) * 100

Common pitfalls

• The standard Top-One test uses equal weighting for all 12 categories, which does not reflect the skewed distribution of trigger words vs. silence in real-world applications.
• Streaming evaluation requires a configurable time tolerance (default 750ms) to match predictions to ground truth timestamps; ignoring this tolerance causes artificially low match rates.
• Open-world categories ('Unknown Word' and 'Silence') must be included in the test set; excluding them inflates accuracy and fails to measure real-world robustness.

Evidence (verbatim from paper)

The standard chosen for the TensorFlow speech commands example code is to look for the ten words "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off", "Stop", and "Go", and have one additional special label for "Unknown Word", and another for "Silence" (no speech detected). The testing is then done by providing equal numbers of examples for each of the twelve categories... If you want to calculate the canonical Top-One error for a model, run inference on each audio clip, and compare the top predicted class against the ground truth label encoded in its containing subfolder name. The proportion of correct predictions will give you the Top-One error.

Citation

@misc{warden2018speechcommands,
  title={Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition},
  author={Pete Warden},
  year={2018},
  note={arXiv:1804.03209}
}

• arXiv: 1804.03209