Mnv 17 Eval

mnv-17-eval

MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech — Mai et al. (2025) (arXiv:2509.18196, 2025)

What this evaluates

Evaluates the ability of speech recognition models to jointly transcribe Mandarin speech and identify nonverbal vocalizations (NVs) like laughs or sighs. It also isolates the strict accuracy of NV event detection and measures whether adding NV recognition degrades core lexical transcription performance.

Datasets

• MNV-17 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• CER (primary) — range: percent
  • Character Error Rate calculated after removing punctuation and spaces from transcripts. Each nonverbal vocalization (NV) label is treated as a single character. Averaged across all test samples.
• NV Recognition Accuracy — range: percent
  • Strict exact-match accuracy. A prediction is correct only if the type, count, and sequential order of all NV events exactly match the ground truth.

Input / output format

Input: Mandarin speech audio recordings containing spoken text and nonverbal vocalizations.

Output: Text transcript containing lexical words and NV labels treated as single characters, plus explicit identification of NV event type, count, and sequential order.

Scoring recipe

def compute_cer(pred, gold):
    pred_clean = remove_punctuation_and_spaces(pred)
    return calculate_cer(pred_clean, gold)

def compute_nv_accuracy(pred, gold):
    pred_nvs = extract_nv_events(pred)
    gold_nvs = extract_nv_events(gold)
    correct = (pred_nvs.type == gold_nvs.type) and (pred_nvs.count == gold_nvs.count) and (pred_nvs.order == gold_nvs.order)
    return int(correct)

Common pitfalls

• Treating NV labels as multiple characters or omitting them during CER calculation, which distorts error rates.
• Using partial matching for NV recognition (e.g., matching only type) instead of the required strict exact match on type, count, and order.
• Failing to strip NV tags from model predictions before calculating pure ASR CER in comparative analyses.

Evidence (verbatim from paper)

Following standard practice [[3], [40]], punctuation and spaces were removed from the transcripts before calculating the Character Error Rate (CER). Each NV label was treated as a single character, and the final CER was averaged across all samples in the test set.

Citation

@misc{mai2025mnv17,
  title={MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech},
  author={Mai et al. (2025)},
  year={2025},
  note={arXiv:2509.18196}
}

• arXiv: 2509.18196