Sytts Commands Eval

sytts-commands-eval

SynTTS-Commands: A Public Dataset for On-Device KWS via TTS-Synthesized Multilingual Speech — Gan et al. (2025) (arXiv:2511.07821, 2025)

What this evaluates

Evaluates the quality and effectiveness of a synthetic multilingual voice command dataset for on-device keyword spotting. It probes whether TTS-synthesized audio can support high-accuracy classification across different model complexities and languages (English and Chinese).

Datasets

• SYNTTS-COMMANDS — total ?; splits: train (-1), test (-1)

Metrics

• classification accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted command labels out of the total number of test instances. Calculated as correct predictions divided by total predictions for multi-class classification.
• cross-entropy loss — range: [0, ∞)
  • Standard categorical cross-entropy loss measuring the divergence between predicted probability distributions and true labels. Lower values indicate better predictive confidence and calibration.

Input / output format

Input: Audio recordings of voice commands in English or Chinese, converted into acoustic features for classification models.

Output: Discrete command class labels (one per audio instance).

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = (predictions == gold_labels).mean()
    loss = categorical_cross_entropy(predictions, gold_labels)
    return {'accuracy': accuracy, 'loss': loss}

Common pitfalls

• Parameter counts vary between English and Chinese deployments due to different output class sizes, not architecture changes, which can mislead size comparisons.
• Lightweight models show significant accuracy drops on Chinese due to tonal/phonetic complexity, which may be mistaken for dataset quality issues rather than model capacity limits.
• Higher accuracy does not always correlate with lower loss; models like EfficientNet-B0 show better calibration despite slightly higher loss.

Evidence (verbatim from paper)

Performance was measured using classification accuracy and cross-entropy loss, enabling a holistic assessment of both predictive power and calibration across languages and model sizes.

Citation

@misc{gan2025syttscommands,
  title={SynTTS-Commands: A Public Dataset for On-Device KWS via TTS-Synthesized Multilingual Speech},
  author={Gan et al. (2025)},
  year={2025},
  note={arXiv:2511.07821}
}

• arXiv: 2511.07821