Xtreme S Eval

xtreme-s-eval

XTREME-S: Evaluating Cross-lingual Speech Representations — Conneau et al. (2022) (arXiv:2203.10752, 2022)

What this evaluates

Evaluates cross-lingual speech representations across 102 languages and four task families: automatic speech recognition, speech translation, speech classification, and speech-text retrieval. It probes the ability of self-supervised speech models to generalize across languages, domains, and varying data regimes from high-resource to low-resource settings.

Datasets

• Fleurs — total ?; splits: train (-1), dev (-1), test (-1)
• MLS — total ?; splits: train (-1)
• VoxPopuli — total ?; splits: train (-1)
• CoVoST-2 — total ?; splits: train (-1), dev (-1), test (-1)
• Minds-14 — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of word edits (insertions, deletions, substitutions) required to change the predicted transcript into the reference transcript, normalized by the total number of words in the reference. Lower is better.
• BLEU — range: other
  • Bilingual Evaluation Understudy: geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty to discourage overly short translations. Higher is better.
• Accuracy — range: percent
  • Percentage of correctly predicted class labels out of the total number of instances. Higher is better.
• F1 — range: percent
  • Harmonic mean of precision and recall across all classes, typically macro-averaged for multi-class tasks. Higher is better.
• P@1 — range: percent
  • Precision at rank 1: the fraction of queries where the correct target transcription or translation appears as the top-1 result in a retrieval pool. Higher is better.

Input / output format

Input: Raw audio waveform (speech utterance) for recognition, translation, and retrieval tasks; audio waveform for classification tasks.

Output: Predicted text transcript (ASR), predicted translated text (ST), predicted class label (classification), or ranked list of candidate transcriptions/translations from a retrieval pool (retrieval).

Scoring recipe

def compute_metrics(predictions, references):
    # WER
    wer_scores = [wer(ref, pred) for pred, ref in zip(predictions['asr'], references['asr'])]
    # BLEU
    bleu_score = corpus_bleu(references['st'], predictions['st'])
    # Accuracy
    acc = sum(1 for p, r in zip(predictions['cls'], references['cls']) if p == r) / len(references['cls'])
    # F1
    f1 = f1_score(references['cls'], predictions['cls'], average='macro')
    # P@1 (Retrieval)
    p1 = sum(1 for q in queries if references[q] == predictions['retrieval'][q][0]) / len(queries)
    return {'WER': mean(wer_scores), 'BLEU': bleu_score, 'Accuracy': acc, 'F1': f1, 'P@1': p1}

Common pitfalls

• Overfitting to speaker identity in Fleurs-LID due to train/dev/test speaker mismatch, which artificially inflates language identification accuracy.
• Using additional supervision beyond the CoVoST-2 dataset for speech translation, which violates the benchmark's constraint to report results leveraging only the provided supervision.
• Running classification tasks with only a single fine-tuning run, leading to high variance; the authors recommend averaging over multiple seeds.

Evidence (verbatim from paper)

In Table 3, we report average character and word error rates on Fleurs, MLS and VoxPopuli. We see that mSLAM obtains the best performance on MLS and VoxPopuli with 9.7 and 9.1 average WER.

Citation

@misc{conneau2022xtremes,
  title={XTREME-S: Evaluating Cross-lingual Speech Representations},
  author={Conneau et al. (2022)},
  year={2022},
  note={arXiv:2203.10752}
}

• arXiv: 2203.10752