Paddlespeech Eval

paddlespeech-eval

PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit — Zhang et al. (2022) (arXiv:2205.12007, 2022)

What this evaluates

Evaluates a speech processing toolkit across five core tasks: environmental sound classification, automatic speech recognition, punctuation restoration, speech translation, and text-to-speech synthesis. It probes the model's ability to handle diverse audio and text inputs, perform sequence labeling, and generate high-quality synthetic speech.

Datasets

• ESC-50 — total 2000; splits: 5-fold cross-validation (-1)
• Librispeech — total ?; splits: train (-1), val (-1), test-clean (-1), test-other (-1); repo http://www.openslr.org/12/
• Aishell-1 — total ?; splits: train (340), val (40), test (20); repo http://www.aishelltech.com/kysjcp
• IWSLT2012-zh — total 150000; splits: train (147000), val (2000), test (1000); repo https://hltc.cs.ust.hk/iwslt/
• MuST-C — total ?; splits: tst-COMMON (-1)
• CSMSC — total ?; splits: train (-1); repo https://www.data-baker.com/open_source.html

Metrics

• 5-fold cross validation accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified audio samples out of the total, averaged over 5 random data splits.
• WER — range: [0, 1]
  • Word Error Rate: normalized edit distance between predicted and reference word sequences.
• CER — range: [0, 1]
  • Character Error Rate: normalized edit distance between predicted and reference character sequences.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for sequence labeling tasks (comma, period, question mark, empty).
• BLEU — range: [0, 100]
  • Detokenized case-sensitive BLEU score using SacreBLEU, measuring n-gram overlap between translated and reference text.
• MOS — range: [1, 5]
  • Mean Opinion Score: average rating from human listeners on a 1-5 scale for naturalness, collected via crowdMOS toolkit.

Input / output format

Input: Audio waveform files (resampled to 32kHz, monophonic) or text sequences, depending on the specific task.

Output: Predicted class labels, transcribed text, restored punctuation tags, translated text, or synthesized audio waveforms.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'sound_classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task == 'asr':
        return edit_distance(predictions, gold) / len(gold)  # WER/CER
    elif task == 'punctuation':
        return f1_score(predictions, gold)
    elif task == 'translation':
        return sacrebleu.compute(predictions, [gold])
    elif task == 'tts':
        return human_mos_ratings(predictions)  # 1-5 scale
    return metric_value

Common pitfalls

• Librispeech results are reported separately for test-clean and test-other splits; mixing them inflates performance.
• TTS evaluation relies on subjective human MOS scores rather than objective acoustic metrics, making cross-study comparison difficult.
• Punctuation restoration simplifies the target set to only comma, period, and question mark, ignoring other punctuation types.

Evidence (verbatim from paper)

We report word error rate (WER) and character error rate (CER) for Librispeech (English) and Aishell (Mandarin) speech recognition, respectively.

Citation

@misc{zhang2022paddlespeech,
  title={PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2205.12007}
}

• arXiv: 2205.12007