Omnilingual Asr Eval

omnilingual-asr-eval

Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages — Keren et al. (2025) (arXiv:2511.09690, 2025)

What this evaluates

Evaluates multilingual automatic speech recognition (ASR) capabilities across 1,600+ languages, including zero-shot generalization to previously unsupported languages. It measures transcription accuracy under varying resource conditions and compares performance against established baselines like Whisper, USM, and MMS.

Datasets

• MMS-Lab — total ?; splits: dev (-1), test (-1)
• FLEURS — total ?; splits: dev (-1), test (-1)
• MLS — total ?; splits: dev (-1), test (-1)
• Common Voice 22 — total ?; splits: dev (-1), test (-1)

Metrics

• CER (primary) — range: percent
  • Character Error Rate: the minimum number of character edits (substitutions, deletions, insertions) required to transform the predicted transcript into the reference transcript, divided by the number of characters in the reference.
• WER — range: percent
  • Word Error Rate: identical to CER but computed at the word level instead of the character level.
• Win Rate — range: percent
  • Percentage of languages on which the evaluated model achieves a lower CER than the baseline model on a given benchmark.

Input / output format

Input: Raw audio utterances, optionally conditioned with source and target language identifier (LID) tokens for translation or script-specific decoding.

Output: Transcribed text sequence (character-level for CTC models, token-level for LLM-ASR models).

Scoring recipe

def compute_avg_cer(predictions_by_lang, references_by_lang):
    lang_cers = []
    for lang in predictions_by_lang:
        preds = predictions_by_lang[lang]
        refs = references_by_lang[lang]
        total_errors = sum(levenshtein(p, r) for p, r in zip(preds, refs))
        total_chars = sum(len(r) for r in refs)
        lang_cers.append(total_errors / total_chars if total_chars > 0 else 0.0)
    return sum(lang_cers) / len(lang_cers)

Common pitfalls

• Averaging CER across languages rather than across utterances, which disproportionately weights low-resource languages and can mask performance on high-resource ones.
• CTC models frequently suffer from script misprediction in low-resource settings, outputting characters from entirely different languages when the wrong script is selected.
• LM fusion significantly reduces CER on FLEURS and MLS benchmarks but is not applied uniformly across all model variants or baseline comparisons.

Evidence (verbatim from paper)

We report character error rate (CER) averaged across languages. In this comparison, we only considered languages that Whisper covers in each benchmark; the number following each dataset name indicates the corresponding number of languages evaluated.

Citation

@misc{keren2025omnilingualasr,
  title={Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages},
  author={Keren et al. (2025)},
  year={2025},
  note={arXiv:2511.09690}
}

• arXiv: 2511.09690