Gigaspeech2 Asr Eval

gigaspeech2-asr-eval

GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement — Yang et al. (2024) (arXiv:2406.11546, 2024)

What this evaluates

Evaluates automatic speech recognition (ASR) models on low-resource languages (Thai, Indonesian, Vietnamese) to measure transcription accuracy against reference texts. It probes the model's ability to handle domain-shifted audio and varying linguistic structures using character-level or word-level error metrics.

Datasets

• GigaSpeech 2 — total ?; splits: DEV (-1), TEST (-1)
• Common Voice 17.0 — total ?; splits: TEST (-1)
• FLEURS — total ?; splits: TEST (-1)

Metrics

• CER/WER (primary) — range: percent
  • Standard Levenshtein edit distance normalized by the reference length. Character-level edits are counted for Thai (CER); word-level edits are counted for Indonesian and Vietnamese (WER).

Input / output format

Input: Raw audio recordings of speech in Thai, Indonesian, or Vietnamese.

Output: Transcribed text string corresponding to the input audio.

Scoring recipe

def compute_cer_wer(predictions, references, word_level=False):
    total_errors = 0
    total_ref_len = 0
    for p, r in zip(predictions, references):
        tokens_p = p.split() if word_level else list(p)
        tokens_r = r.split() if word_level else list(r)
        total_errors += levenshtein_distance(tokens_p, tokens_r)
        total_ref_len += len(tokens_r)
    return (total_errors / total_ref_len) * 100

Common pitfalls

• CER is used for Thai while WER is used for Indonesian and Vietnamese; mixing these up will invalidate cross-lingual comparisons.
• Performance drops on Common Voice and FLEURS are attributed to domain mismatch rather than model failure, so cross-dataset comparisons should account for audio distribution shifts.
• Comparing models with vastly different parameter counts (e.g., 151.9M vs 1.5B) without noting compute/size differences can be misleading.

Evidence (verbatim from paper)

The ASR performance is evaluated regarding character error rate (CER) or word error rate (WER) on three distinct test sets from GigaSpeech 2, Common Voice 17.0, and FLEURS.

Citation

@misc{yang2024gigaspeech2,
  title={GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.11546}
}

• arXiv: 2406.11546