Wildasr Eval

wildasr-eval

Back to Basics: Revisiting ASR in the Age of Voice Agents — Tay et al. (2026) (arXiv:2603.25727, 2026)

What this evaluates

This benchmark probes the robustness of automatic speech recognition (ASR) systems under realistic, out-of-distribution conditions. It specifically evaluates performance degradation across environmental noise, demographic shifts (accent, age, child speech), and linguistic diversity (short, incomplete, code-switched utterances), while also measuring semantic hallucination rates beyond standard lexical error metrics.

Datasets

• WildASR — total ?; splits: test (-1); repo https://github.com/boson-ai/WildASR-public

Metrics

• WER/CER (primary) — range: percent
  • Word Error Rate (WER) or Character Error Rate (CER) calculated as the minimum number of insertions, deletions, and substitutions required to transform the predicted transcript into the ground truth, divided by the total number of words/characters in the reference. Reported as a percentage.
• HER — range: percent
  • Hallucination Error Rate measures semantic-level errors and fabricated content that standard lexical metrics miss. It quantifies the proportion of utterances where the model generates plausible but unspoken content or meaning-altering hallucinations, particularly under short, incomplete, or code-switched inputs.

Input / output format

Input: Raw audio recordings of human speech, perturbed with environmental degradations (noise, reverberation), demographic shifts (accents, child/older speakers), or linguistic variations (short, truncated, code-switched).

Output: A single string containing the transcribed text output by the ASR model.

Scoring recipe

def compute_metrics(predictions, references):
    wer_scores = [wer(reference, pred) for pred, reference in zip(predictions, references)]
    cer_scores = [cer(reference, pred) for pred, reference in zip(predictions, references)]
    # HER follows Atwany et al. (2025) protocol for semantic hallucination detection
    her_scores = [1.0 if is_hallucination(pred, reference) else 0.0 for pred, reference in zip(predictions, references)]
    return {
        "WER": sum(wer_scores) / len(wer_scores) * 100,
        "CER": sum(cer_scores) / len(cer_scores) * 100,
        "HER": sum(her_scores) / len(her_scores) * 100
    }

Common pitfalls

• Relying solely on corpus-level mean WER/CER masks severe tail failures; the P90 elbow metric is required to identify instability thresholds for deployment.
• Using only lexical metrics (WER/CER) obscures critical semantic hallucinations; HER must be computed alongside lexical error rates to detect meaning-altering fabrications.
• Assuming robustness transfers across conditions; performance on clean/read speech does not predict behavior under noise, demographic shifts, or short utterances.

Evidence (verbatim from paper)

To understand hallucination behavior, we compute Hallucination Error Rate (HER) (Atwany et al., 2025) to assess semantic-level errors beyond lexical metrics. Discrepancies between WER/CER and HER highlight cases where surface-level transcription appears reasonable despite severe meaning distortion.

Citation

@misc{tay2026back,
  title={Back to Basics: Revisiting ASR in the Age of Voice Agents},
  author={Tay et al. (2026)},
  year={2026},
  note={arXiv:2603.25727}
}

• arXiv: 2603.25727