Gigaspeech Eval

gigaspeech-eval

GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio — Chen et al. (2021) (arXiv:2106.06909, 2021)

What this evaluates

Evaluates automatic speech recognition (ASR) systems on a large-scale, multi-domain English corpus containing both read and spontaneous speech. It benchmarks transcription accuracy across tiered training subsets and professionally re-transcribed evaluation sets using word error rate.

Datasets

• GigaSpeech — total 10000; splits: train (-1), dev (-1), test (-1); repo https://github.com/SpeechColab/GigaSpeech

Metrics

• WER (primary) — range: percent
  • Word Error Rate, calculated as the number of insertions, deletions, and substitutions required to transform the predicted transcription into the reference transcription, divided by the total number of words in the reference.

Input / output format

Input: Audio recordings (typically processed into acoustic features such as 80-dim log Fbanks)

Output: Text transcription corresponding to the audio input

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Baseline results are toolkit-specific and explicitly stated as not suitable for cross-toolkit performance comparison.
• The XS training subset (10 hours) is intended only for system building and debugging, not for evaluating strong model performance.
• Evaluation sets (DEV and TEST) are professionally re-transcribed to ensure high fidelity, unlike training data which relies on forced alignment with strict WER caps.

Evidence (verbatim from paper)

Table 4: GigaSpeech baselines for the XL training subset (WER in %) ... Table 5: Kaldi baselines for GigaSpeech training subsets (WER in %)

Citation

@misc{chen2021gigaspeech,
  title={GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2106.06909}
}

• arXiv: 2106.06909