Wilder Eval

wilder-eval

WildElder: A Chinese Elderly Speech Dataset from the Wild with Fine-Grained Manual Annotations — Wang et al. (2025) (arXiv:2510.09344, 2025)

What this evaluates

This benchmark evaluates automatic speech recognition (ASR) capabilities on Mandarin speech produced by elderly individuals. It probes a model's robustness to real-world acoustic degradation, articulation variability, tremors, and diverse accent strengths under uncontrolled recording conditions.

Datasets

• WildElder — total 23701; splits: train (18835), dev (2465), test (2400); repo https://github.com/NKU-HLT/WildElder

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Standard ASR metric measuring the percentage of words incorrectly recognized relative to the ground truth transcript. Calculated as (Substitutions + Deletions + Insertions) / Total Words in reference.

Input / output format

Input: Raw audio recordings of Mandarin speech (average ~5.1 seconds per utterance) collected from online videos.

Output: Predicted text transcript corresponding to each input audio utterance.

Scoring recipe

def compute_wer(predictions, references):
    total_words = 0
    total_errors = 0
    for pred, ref in zip(predictions, references):
        ref_words = ref.split()
        pred_words = pred.split()
        total_words += len(ref_words)
        edits = levenshtein_distance(pred_words, ref_words)
        total_errors += edits
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Splits are performed at the speaker level, not utterance level, so standard random splitting will cause severe data leakage.
• The dataset contains 'in-the-wild' recordings with uncontrolled background noise, tremors, and varying accent strengths, making lab-clean baselines misleading.
• Mandarin speech is character-based; using word-level tokenization without proper Chinese word segmentation will inflate error rates.

Evidence (verbatim from paper)

To ensure reproducibility, WildElder is divided into training, development, and test sets at the speaker level. The training set contains 18,835 utterances (26.7 h), the development set 2,465 utterances (3.5 h), and the test set 2,400 utterances (3.5 h). The average utterance duration is around 5.1 seconds across all subsets, showing a balanced partition suitable for training and evaluation.

Citation

@misc{wang2025wilder,
  title={WildElder: A Chinese Elderly Speech Dataset from the Wild with Fine-Grained Manual Annotations},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.09344}
}

• arXiv: 2510.09344