Common Voice Asr Eval

common-voice-asr-eval

Common Voice: A Massively-Multilingual Speech Corpus — Ardila et al. (2019) (arXiv:1912.06670, 2019)

What this evaluates

Evaluates multilingual automatic speech recognition (ASR) capabilities, specifically testing speaker generalization and low-resource language adaptation via transfer learning from an English model. It measures how well a model can transcribe audio from diverse, crowdsourced speakers across multiple languages with varying data sizes.

Datasets

• Common Voice — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• character error rate (primary) — range: percent
  • Standard character-level edit distance (Levenshtein) between the predicted transcription and the ground truth text, normalized by the length of the reference text.

Input / output format

Input: Audio clips in MPEG-3 format, processed into 494-dimensional vectors (19 spliced frames of 26 MFCC features each), paired with ground truth text transcriptions.

Output: Predicted character sequence corresponding to the input audio clip.

Scoring recipe

def compute_cer(predictions, golds):
    total_errors = 0
    total_refs = 0
    for pred, gold in zip(predictions, golds):
        errors = levenshtein_distance(pred, gold)
        total_errors += errors
        total_refs += len(gold)
    return total_errors / total_refs

Common pitfalls

• Splits enforce strict speaker separation (one speaker only in one split) to test speaker generalization, resulting in very small training sets per language.
• Early stopping uses a custom criterion based on a 5-epoch window and a mean loss threshold of 0.5 with std < 0.5, rather than standard patience.
• Transfer learning setup copies layers from a pre-trained English model and initializes new target-language layers, which must be fine-tuned jointly.

Evidence (verbatim from paper)

The corpus enables end-to-end ASR experiments, achieving an average 5.99% character error rate improvement across 12 low-resource languages through transfer learning from an English model... We made dataset splits (c.f. Table (2)) such that one speaker's recordings are only present in one data split. This allows us to make a fair evaluation of speaker generalization... The splits per language were made as close as possible to 80% train, 10% development, and 10% test.

Citation

@misc{ardila2019commonvoice,
  title={Common Voice: A Massively-Multilingual Speech Corpus},
  author={Ardila et al. (2019)},
  year={2019},
  note={arXiv:1912.06670}
}

• arXiv: 1912.06670