Open Asr Leaderboard Eval

open-asr-leaderboard-eval

Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual Speech Recognition Evaluation — Srivastav et al. (2025) (arXiv:2510.06961, 2025)

What this evaluates

Evaluates multilingual speech recognition models on transcription accuracy and inference efficiency across short-form, long-form, and diverse language settings. It standardizes text normalization and reports both word error rate and inverse real-time factor to enable fair accuracy–efficiency comparisons.

Datasets

• Short-form English — total ?; splits: test (-1)
• Multilingual — total ?; splits: test (-1)
• Long-form — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Words. Standard ASR metric measuring transcription accuracy.
• RTFx — range: other
  • Inverse Real-Time Factor: Audio Duration / Inference Time. Measures throughput efficiency (words processed per real-time second).

Input / output format

Input: Raw audio files (short-form, long-form, or multilingual recordings).

Output: Transcribed text string.

Scoring recipe

def compute_wer(predictions, references):
    # Standard WER calculation over tokenized text
    return wer_score(predictions, references) * 100

def compute_rtf(audio_durations, inference_times):
    # RTFx = 1 / RT = audio_duration / inference_time
    return audio_durations / inference_times

Common pitfalls

• Closed-source models cannot be fairly evaluated for RTFx due to upload latency and lack of GPU control.
• Improving English WER often degrades multilingual coverage due to training data specialization.
• SSL encoders paired with CTC decoders significantly underperform compared to Conformer+LLM architectures.

Evidence (verbatim from paper)

While the latter can achieve superior RTFx, this comes at the cost of accuracy: e.g., the best CTC-based model (NVIDIA Parakeet CTC 1.1B) ranks only 23rd in terms of WER.

Citation

@misc{srivastav2025openasr,
  title={Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual Speech Recognition Evaluation},
  author={Srivastav et al. (2025)},
  year={2025},
  note={arXiv:2510.06961}
}

• arXiv: 2510.06961