Cs Dialogue Asr Eval

cs-dialogue-asr-eval

CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition — Jiaming Zhou et al. (2025) (arXiv:2502.18913, 2025)

What this evaluates

This benchmark evaluates automatic speech recognition (ASR) systems on their ability to accurately transcribe spontaneous, full-length dialogues that alternate between Mandarin and English. It probes a model's robustness to language alternation, phonetic mismatches, and contextual dependencies in naturalistic code-switching scenarios.

Datasets

• CS-Dialogue — total 104; splits: train (-1), test (-1)

Metrics

• MER (primary) — range: percent
  • Mixture Error Rate (also called Mixed Error Rate). It calculates the edit distance between the reference and hypothesis transcriptions, treating both Chinese characters and English words as atomic units for alignment and error counting.
• WER — range: percent
  • Word Error Rate. Standard ASR metric calculating the minimum number of word insertions, deletions, and substitutions required to transform the predicted text into the reference, divided by the total number of words in the reference.
• CER — range: percent
  • Character Error Rate. Standard ASR metric calculating the minimum number of character insertions, deletions, and substitutions required to transform the predicted text into the reference, divided by the total number of characters in the reference.

Input / output format

Input: Audio recordings of spontaneous Mandarin-English code-switching dialogues.

Output: Text transcription containing both Mandarin characters and English words.

Scoring recipe

def compute_asr_metrics(reference, prediction):
    ref_tokens = tokenize_mixed(reference)  # split English by space, keep Chinese chars
    pred_tokens = tokenize_mixed(prediction)
    n_ref = len(ref_tokens)
    if n_ref == 0: return {'CER': 0, 'WER': 0, 'MER': 0, 'S': 0, 'D': 0, 'I': 0}
    ops = levenshtein(ref_tokens, pred_tokens)
    n_sub, n_del, n_ins = ops['sub'], ops['del'], ops['ins']
    cer = (n_sub + n_del + n_ins) / n_ref * 100
    wer = cer
    mer = (n_sub + n_del + n_ins) / n_ref * 100
    return {'CER': cer, 'WER': wer, 'MER': mer, 'S': n_sub, 'D': n_del, 'I': n_ins}

Common pitfalls

• Code-switching introduces phonetic mismatches and language alternation that standard monolingual ASR tokenizers often mishandle, leading to inflated error rates if not properly segmented.
• Model performance varies significantly by conversation topic (e.g., 'Sports' and 'Philosophy' yield higher MER), so reporting a single aggregate metric without topic breakdown can mask domain-specific weaknesses.
• Substitution errors dominate over deletions and insertions across all models, meaning metrics focusing only on insertion/deletion penalties may misrepresent the actual error profile.

Evidence (verbatim from paper)

ASR performance on the code-switching dataset is evaluated using three metrics: Mixture Error Rate (MER), Word Error Rate (WER), and Character Error Rate (CER). Following Shi et al. ([2020]), MER is adopted as the primary metric due to its holistic assessment of ASR accuracy, calculating the edit distance considering both Chinese characters and English words.

Citation

@misc{zhou2025csdialogue,
  title={CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition},
  author={Jiaming Zhou et al. (2025)},
  year={2025},
  note={arXiv:2502.18913}
}

• arXiv: 2502.18913