Contextasr Bench Eval

contextasr-bench-eval

ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark — He Wang et al. (2025) (arXiv:2507.05727, 2025)

What this evaluates

Evaluates how well ASR models and Large Audio Language Models (LALMs) leverage contextual world knowledge and linguistic reasoning to transcribe speech containing named entities. It tests performance across ten domains under three context conditions: no context, coarse-grained domain labels, and fine-grained technical terms.

Datasets

• ContextASR-Bench — total 40000; splits: test (-1); repo https://github.com/MrSupW/ContextASR-Bench

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Calculated as the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcription into the ground-truth text, divided by the total number of words in the reference.

Input / output format

Input: Audio recordings of synthetic speech (generated via TTS from domain-specific text seeds) paired with context prompts (contextless, coarse-grained domain labels, or fine-grained technical terms).

Output: Text transcription of the spoken audio.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Models may over-rely on coarse context without properly aligning it to acoustic features, leading to inflated performance on domain labels but poor transcription accuracy.
• Synthetic TTS speech may introduce artifacts that disproportionately affect conventional ASR models compared to LALMs, skewing cross-model comparisons.
• Fine-grained context prompts must be carefully formatted to avoid directly leaking answer tokens, which would artificially lower error rates.

Evidence (verbatim from paper)

To highlight our proposed ContextASR-Bench in assessing how LLMs' world knowledge and context modeling capabilities enhance contextual speech recognition, we conduct comprehensive evaluations. We evaluate conventional ASR models, including Paraformer-Large (Gao et al. 2022), SenseVoice-Small (An et al. 2024), Whisper-Largev3 and turbo (Radford et al. 2023), FireredASR-AED-L and FireredASR-LLM-L (Xu et al. 2025b), Dolphin-Base and Small (Meng et al. 2025), as well as LALMs, including Qwen2-Audio (Chu et al. 2024), Qwen2.5-Omni (Xu et al. 2025a), Baichuan-Audio (Li et al. 2025a), Baichuan-Omni1.5 (Li et al. 2025b), and Kimi-Audio (KimiTeam et al. 2025). All user prompts for LALMs under three context evaluation settings can be found in Appendix D.

Citation

@misc{wang2025contextasrbench,
  title={ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark},
  author={He Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.05727}
}

• arXiv: 2507.05727