Nim4 Asr Eval

nim4-asr-eval

NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR — Yuan Xie et al. (2026) (arXiv:2604.18105, 2026)

What this evaluates

Evaluates automatic speech recognition performance across diverse acoustic and linguistic domains, including English, Mandarin, dialects, code-switching, and in-car conversational scenarios. It measures transcription accuracy and hallucination rates to assess model robustness, latency, and customization capabilities.

Datasets

• LibriSpeech — total ?; splits: dev (-1), test (-1)
• VoxPopuli — total ?; splits: dev (-1), test (-1)
• MLS-English — total ?; splits: test (-1)
• AISHELL-1 — total ?; splits: dev (-1), test (-1)
• AISHELL-2 — total ?; splits: ios (-1), test (-1)
• AISHELL-2021-Eval — total ?; splits: A (-1), C (-1), D (-1)
• WeNetSpeech — total ?; splits: meeting (-1), net (-1)
• SpeechIO — total ?; splits: test (-1)
• WeNetSpeech-Chuan — total ?; splits: easy (-1), hard (-1)
• WeNetSpeech-Yue — total ?; splits: short (-1), long (-1)
• KeSpeech — total ?; splits: test (-1)
• CS-Dialogue — total ?; splits: test (-1)
• ASCEND — total ?; splits: test (-1)
• M4Singer — total ?; splits: test (-1)
• Internal POI Benchmarks — total ?; splits: City A (-1), City B (-1), City C (-1), City D (-1)
• Internal Media Benchmarks — total ?; splits: Music (-1), Video (-1), Radio (-1)
• Internal Device Control — total ?; splits: Vehicle control (-1)
• Internal Conversational — total ?; splits: Vehicle-domain chat (-1), Multi-domain chat (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: ratio of substitutions, deletions, and insertions to the total number of words in the reference transcription. Used for English benchmarks.
• CER (primary) — range: percent
  • Character Error Rate: ratio of substitutions, deletions, and insertions to the total number of characters in the reference transcription. Used for Mandarin, dialects, lyrics, and code-switched benchmarks.
• Hallucination Rate — range: percent
  • Ratio of hallucinated samples to total samples. A sample is hallucinated if its transcription length exceeds the ground-truth length by over 50% with negligible lexical overlap.
• POI Recall — range: percent
  • Proportion of Point of Interest (POI) entities correctly recognized in the transcription output.

Input / output format

Input: Audio recordings paired with ground-truth transcriptions.

Output: Text transcriptions generated by the ASR model.

Scoring recipe

def compute_wer_cer(pred, ref):
    pred_norm = normalize_text(pred)
    ref_norm = normalize_text(ref)
    ops = levenshtein_operations(pred_norm, ref_norm)
    errors = ops['substitutions'] + ops['deletions'] + ops['insertions']
    total = len(ref_norm.split()) if is_english else len(ref_norm)
    return (errors / total) * 100

def compute_hallucination_rate(predictions, references):
    hallucinated_count = 0
    for pred, ref in zip(predictions, references):
        if len(pred) > len(ref) * 1.5 and lexical_overlap(pred, ref) < threshold:
            hallucinated_count += 1
    return (hallucinated_count / len(predictions)) * 100

Common pitfalls

• Text normalization via WeTextProcessing is mandatory for both predictions and references; skipping it inflates error rates and breaks fair comparison.
• Baselines are strictly evaluated in offline mode, while NIM4-ASR reports both offline and streaming results; comparing streaming NIM4-ASR against offline baselines is invalid.
• Hallucination rate evaluation explicitly excludes WeNetSpeech Meeting, SpeechIO, MLS-English, and WeNetSpeech Net due to zero hallucinations or unreliable annotations.

Evidence (verbatim from paper)

We report Word Error Rate (WER) for English benchmarks, and Character Error Rate (CER) for Mandarin, Chinese dialect, lyrics, and code-switched Chinese-English benchmarks. ... Specifically, a sample is classified as hallucinated if its transcription exceeds the ground-truth length by over 50% with negligible lexical overlap.

Citation

@misc{xie2026nim4asr,
  title={NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR},
  author={Yuan Xie et al. (2026)},
  year={2026},
  note={arXiv:2604.18105}
}

• arXiv: 2604.18105