Voxpopuli Eval

voxpopuli-eval

VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation — Wang et al. (2021) (arXiv:2101.00390, 2021)

What this evaluates

Benchmarks multilingual speech representation learning and semi-supervised ASR/ST performance across multiple languages and domains, measuring phoneme discriminability, recognition accuracy, and translation quality.

Datasets

• VoxPopuli — total 400000; splits: unlabeled (400000), transcribed (1800)
• Common Voice — total ?; splits: train (3600), dev (1200), test (3600)
• ZeroSpeech 2017 — total ?; splits: test (10)
• EuroParl-ST — total ?; splits: train (-1)
• CoVoST 2 — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the ratio of insertions, deletions, and substitutions to the total number of words in the reference transcript. Lower values indicate better performance.
• PER — range: percent
  • Phone Error Rate: identical to WER but computed over phoneme sequences using a phoneme vocabulary. Lower values indicate better performance.
• BLEU — range: percent
  • Bilingual Evaluation Understudy: geometric mean of modified n-gram precisions between predicted and reference translations, with brevity penalty. Higher values indicate better performance.
• ABX discriminability score — range: other
  • Measures phoneme discriminability of unsupervised features by computing the error rate of a linear classifier distinguishing phoneme B from A and X, where A and X share the same phoneme label. Lower values indicate better feature discriminability.

Input / output format

Input: Raw audio utterances (often segmented by VAD and speaker diarized), optionally paired with transcribed text for fine-tuning or decoding. Features may be 80-dim log-mel filterbanks with CMVN or raw samples.

Output: Predicted phoneme sequences, word sequences, or translated text, depending on the task and vocabulary used.

Scoring recipe

def compute_metric(predictions, gold, task):
    if task == 'ASR':
        return edit_distance(predictions, gold) / len(gold)  # PER or WER based on vocab
    elif task == 'ST':
        return bleu_score(gold, predictions)
    elif task == 'ABX':
        return abx_discriminability(audio_features, phoneme_boundaries)
    return None

Common pitfalls

• Domain mismatch: VoxPopuli pre-training data consists of political oral speeches, while fine-tuning/test sets like Common Voice contain read speech, which can skew generalization metrics.
• Checkpoint selection varies by experiment: best validation loss checkpoint is used for standard fine-tuning, but the average of the 10 best checkpoints is used for self-training experiments.
• Vocabulary choice directly impacts metrics: phoneme vocabularies are required for PER evaluation, while character or subword vocabularies are used for WER and BLEU.

Evidence (verbatim from paper)

We follow the setting in Rivière et al. (2020) to evaluate unsupervised speech representations by phoneme discriminability on 3 languages (English, French and Mandarin), and report ABX discriminability score on the 10s test set from ZeroSpeech 2017. ... We report test WER on Common Voice (CV). ... Left: test BLEU for ST models. Right: test WER for ASR models.

Citation

@misc{wang2021voxpopuli,
  title={VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2101.00390}
}

• arXiv: 2101.00390