Libriheavy Asr Eval

libriheavy-asr-eval

Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context — Wei Kang et al. (arXiv:2309.08105, 2023)

What this evaluates

Evaluates automatic speech recognition (ASR) models on long-form audio, measuring accuracy in predicting word and character sequences. It specifically probes the model's ability to handle full-text formatting, including punctuation and casing, and tests performance across different training data scales.

Datasets

• Libriheavy — total ?; splits: lh-clean (-1), lh-other (-1); repo https://github.com/k2-fsa/libriheavy

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the number of insertions, deletions, and substitutions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Expressed as a percentage.
• CER — range: percent
  • Character Error Rate: the number of insertions, deletions, and substitutions required to transform the predicted transcript into the reference transcript, divided by the total number of characters in the reference. Expressed as a percentage.

Input / output format

Input: 80-channel Fbank audio features extracted with 25 ms windows shifted by 10 ms, with dither=0.1. Models may also receive preceding text context.

Output: Text transcript. Evaluated on both normalized text (uppercase, no punctuation, expanded numbers/abbreviations) and full-format text (with punctuation and casing).

Scoring recipe

def wer_cer(pred_str, ref_str, char_mode=False):
    pred_tokens = pred_str.split() if not char_mode else list(pred_str)
    ref_tokens = ref_str.split() if not char_mode else list(ref_str)
    dist = edit_distance(pred_tokens, ref_tokens)
    total = len(ref_tokens)
    return (dist / total) * 100.0 if total > 0 else 0.0

Common pitfalls

• Hypotheses must undergo text normalization (e.g., converting numbers and abbreviations to their textual forms) before comparison to match the reference transcripts.
• Evaluating punctuation and casing requires a larger BPE vocabulary (756-class vs 500-class) and explicit fallback byte handling for rare characters.
• Performance gaps between normalized and unnormalized training texts shrink significantly as training data size increases, masking early-stage formatting benefits.

Evidence (verbatim from paper)

Table 2 shows the Word Error Rate (WER) of the models on Libriheavy test sets. As a reference, we also show the WER on the LibriSpeech test sets. The N-Best hypotheses are first generated by the CTC branch and then rescored by the attention branch. Note that for the LibriSpeech results, we apply some simple text normalization, such as converting numbers to their corresponding text and converting abbreviations (e.g “Mr.” to “Mister”) on the hypotheses to make it compatible with the LibriSpeech transcripts. We also apply these normalization procedures in the following experiments.

Citation

@misc{kang2023libriheavy,
  title={Libriheavy: a 50,000 hours ASR corpus with punctuation casing and context},
  author={Wei Kang et al.},
  year={2023},
  note={arXiv:2309.08105}
}

• arXiv: 2309.08105