Muscat Eval

muscat-eval

MUSCAT: MUltilingual, SCientific ConversATion Benchmark — Sinhamahapatra et al. (2026) (arXiv:2604.15929, 2026)

What this evaluates

Evaluates multilingual automatic speech recognition (ASR) systems on spontaneous scientific conversations, focusing on their ability to handle code-switching, transcribe domain-specific technical terms, and maintain accuracy across varying audio recording devices and segmentation methods.

Datasets

• MUSCAT — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Standard Word Error Rate computed as the total number of substitutions, deletions, and insertions normalized by the length of the reference transcript. Gives equal importance to every word.
• WER_t_ref — range: percent
  • Reference-centric domain-specific WER: (|substituted| + |deleted|) / (|recognized| + |substituted| + |deleted|), calculated only over technical terms extracted from referenced scientific papers.
• WER_t_hyp — range: percent
  • Hypothesis-centric domain-specific WER: (|substituted| + |inserted|) / (|recognized| + |substituted| + |inserted|), calculated only over technical terms in the model's output.

Input / output format

Input: Audio segments (manually or automatically segmented) containing multilingual scientific dialogue.

Output: Text transcript (hypothesis) corresponding to each audio segment.

Scoring recipe

def compute_wer(hyp, ref):
    # hyp and ref are whitespace-separated word lists
    # For Chinese, apply jieba segmentation first
    edits = levenshtein_distance(hyp, ref)
    return (edits['sub'] + edits['del'] + edits['ins']) / len(ref)

def compute_domain_wer(hyp_words, ref_words, domain_terms):
    hyp_dom = [w for w in hyp_words if w in domain_terms]
    ref_dom = [w for w in ref_words if w in domain_terms]
    edits = levenshtein_distance(hyp_dom, ref_dom)
    wer_ref = (edits['sub'] + edits['del']) / len(ref_dom) if ref_dom else 0
    wer_hyp = (edits['sub'] + edits['ins']) / len(hyp_dom) if hyp_dom else 0
    return wer_ref, wer_hyp

Common pitfalls

• Chinese text requires jieba segmentation before WER calculation due to lack of whitespace.
• Automatic segmentation (SHAS) often mixes languages in single segments, artificially inflating WER compared to manual/oracle splits.
• Domain-specific WER only evaluates technical terms extracted from referenced papers, ignoring general vocabulary performance.

Evidence (verbatim from paper)

Word Error Rate (WER) is a common metric used to evaluate the accuracy of ASR systems. It measures how much the transcribed text deviates from the ground truth by computing the number of errors made during transcription, giving equal importance to every word in the transcript. Unlike the other languages in our dataset, Chinese is not a whitespace-separated language. We use jieba, a Python Chinese word segmentation tool for segmenting the Chinese text into words.

Citation

@misc{sinhamahapatra2026muscat,
  title={MUSCAT: MUltilingual, SCientific ConversATion Benchmark},
  author={Sinhamahapatra et al. (2026)},
  year={2026},
  note={arXiv:2604.15929}
}

• arXiv: 2604.15929