qhjqhj00 multilingual-tedx-eval
The Multilingual TEDx Corpus for Speech Recognition and Translation — Salesky et al. (2021) (arXiv:2102.01757, 2021)
What this evaluates
Evaluates automatic speech recognition, machine translation, and speech translation capabilities on a multilingual corpus of TEDx talks. It probes model robustness to lower-resource conditions, cross-lingual transfer, and the effectiveness of cascaded versus end-to-end modeling paradigms.
Datasets
- Multilingual TEDx Corpus — total ?; splits: test (-1); repo https://github.com/m-wiesner/tedx
Metrics
WER— range: percent- Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the number of words in the reference.
BLEU(primary) — range: percent- Cased tokenized BLEU score computed using SACRE-BLEU. Calculated as the geometric mean of modified n-gram precisions with a brevity penalty.
Input / output format
Input: For ASR: 16kHz mono audio files. For MT/ST: source language text sentences (with language ID tags appended at the beginning for multilingual models).
Output: For ASR: predicted word transcript. For MT/ST: translated target language text sentence.
Scoring recipe
def compute_wer(predictions, references):
total_errors = 0
total_refs = 0
for pred, ref in zip(predictions, references):
total_errors += edit_distance(pred.split(), ref.split())
total_refs += len(ref.split())
return (total_errors / total_refs) * 100
def compute_bleu(predictions, references):
return sacrebleu.corpus_bleu(predictions, [references]) * 100
Common pitfalls
- ASR evaluation removes punctuation and lowercases text, while MT/ST evaluation retains original punctuation and case, creating inconsistent preprocessing standards across tasks.
- Arabic WER results are heavily skewed because the corpus uses Modern Standard Arabic transcripts rather than the spoken dialect actually present in the audio.
- Cascaded ST models are evaluated using 1-best ASR transcripts (beam=10), which ignores ASR errors in the downstream MT model and may overestimate real-world ST performance.
Evidence (verbatim from paper)
We computed cased tokenized BLEU scores using SACRE-BLEU [22]. Different community standards for punctuation and case exist between ASR, MT, and ST; while we removed punctuation and lowercased text for ASR evaluation, we did not for translation tasks.
Citation
@misc{salesky2021multilingualtedx,
title={The Multilingual TEDx Corpus for Speech Recognition and Translation},
author={Salesky et al. (2021)},
year={2021},
note={arXiv:2102.01757}
}
- arXiv: 2102.01757