Gsc Speech Commands Eval

gsc-speech-commands-eval

Enhancing Synthetic Training Data for Speech Commands: From ASR-Based Filtering to Domain Adaptation in SSL Latent Space — Quintas et al. (2024) (arXiv:2409.12745, 2024)

What this evaluates

Evaluates keyword spotting models trained on real versus synthetic speech data, measuring how ASR-based filtering of hallucinated synthetic commands affects classification accuracy on the Google Speech Commands dataset.

Datasets

• Google Speech Commands (GSC) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly classified audio commands in the test set. Computed as (number of correct predictions / total number of test instances) * 100.

Input / output format

Input: Audio recordings of spoken short commands (keywords), processed through the MatchboxNet convolutional architecture.

Output: Discrete class label corresponding to the spoken command.

Scoring recipe

correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100
return accuracy

Common pitfalls

• Models are trained on synthetic data but evaluated on real GSC test data, creating a domain gap that heavily impacts accuracy.
• Literature baselines (e.g., [17]) include a silence class during training, while this evaluation explicitly excludes it, making direct accuracy comparisons invalid.
• Results are averaged over 5 random seeds with standard deviation reported; single-run results may not be comparable to literature values.

Evidence (verbatim from paper)

Accuracy values are reported in Table 1. As expected, the best results were obtained with the original GSC training data (Real), with accuracy values above 98% . Results obtained in the Synth. condition, using synthetic data for training, achieved accuracy values of 89 - 90% . The ASR-based filtering technique led to an accuracy gain of over two percentage points on MatchboxNet models, demonstrating that uncontrolled hallucinations in synthetic training data can contaminate the dataset and degrade model performance.

Citation

@misc{quintas2024enhancing,
  title={Enhancing Synthetic Training Data for Speech Commands: From ASR-Based Filtering to Domain Adaptation in SSL Latent Space},
  author={Quintas et al. (2024)},
  year={2024},
  note={arXiv:2409.12745}
}

• arXiv: 2409.12745