Vimedcss Eval

vimedcss-eval

ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark — Nguyen et al. (2026) (arXiv:2602.12911, 2026)

What this evaluates

This benchmark evaluates automatic speech recognition (ASR) models on Vietnamese medical audio containing embedded English terminology. It specifically probes the model's ability to accurately transcribe both the matrix language and code-switched segments, measuring overall transcription quality alongside specialized metrics for code-switched and non-code-switched spans.

Datasets

• ViMedCSS — total 16576; splits: train (11833), val (1714), test (1615), hard (658)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed over the full, normalized output sequence. Calculated as the normalized edit distance between the predicted and ground-truth transcriptions.
• CER — range: percent
  • Character Error Rate computed over the full, normalized output sequence. Calculated as the normalized edit distance at the character level.
• CS-WER — range: percent
  • Word Error Rate computed only on tokens inside code-switched regions. Restricts the edit distance calculation to the subset of tokens marked as code-switched.
• N-WER — range: percent
  • Word Error Rate restricted to tokens that do not require normalization (i.e., outside code-switched spans). Measures accuracy on the matrix language segments.

Input / output format

Input: 16 kHz audio recordings of Vietnamese medical speech with embedded English terms, paired with ground-truth transcriptions and token-level code-switch masks.

Output: Transcribed text string corresponding to the input audio.

Scoring recipe

def compute_metrics(pred, gold, cs_mask):
    # pred, gold: normalized token lists
    # cs_mask: boolean list indicating CS tokens
    wer = edit_distance(pred, gold) / len(gold)
    cer = edit_distance(pred, gold, char_level=True) / len(gold)
    cs_gold = [g for g, is_cs in zip(gold, cs_mask) if is_cs]
    n_gold = [g for g, is_cs in zip(gold, cs_mask) if not is_cs]
    cs_wer = sum(edit_distance(p, g) for p, g in zip(pred, cs_gold)) / len(cs_gold) if cs_gold else 0
    n_wer = sum(edit_distance(p, g) for p, g in zip(pred, n_gold)) / len(n_gold) if n_gold else 0
    return wer, cer, cs_wer, n_wer

Common pitfalls

• Data leakage: The Hard split contains rare CS terms explicitly removed from Train/Valid/Test; including them in training or evaluation without isolation invalidates the generalization test.
• Metric scope confusion: CS-WER and N-WER are computed on disjoint token sets; applying them to the full transcript or mixing them up misrepresents model performance on code-switched vs. matrix language.
• Normalization requirement: WER must be computed on the normalized output sequence, not raw ASR output, as specified in the protocol.

Evidence (verbatim from paper)

Following prior work on Vietnamese code-switching ASR, we report WER and CER together with CS-WER and N-WER to disentangle accuracy on code-switched spans from the rest of the transcript. Concretely, CS-WER is the word error rate computed only on tokens inside code-switched regions, N-WER is the word error rate restricted to tokens that do not require normalization (i.e., outside CS spans), and WER is computed over the full, normalized output sequence.

Citation

@misc{nguyen2026vimedcss,
  title={ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark},
  author={Nguyen et al. (2026)},
  year={2026},
  note={arXiv:2602.12911}
}

• arXiv: 2602.12911