Librispeech Pc Eval

librispeech-pc-eval

LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models — Meister et al. (2023) (arXiv:2310.02943, 2023)

What this evaluates

Evaluates the ability of end-to-end automatic speech recognition (ASR) models to correctly predict punctuation marks and word capitalization in transcribed speech. It specifically isolates punctuation-specific errors to enable fine-grained comparison between cascade and end-to-end architectures.

Datasets

• LibriSpeech-PC — total ?; splits: test (-1)

Metrics

• Punctuation Error Rate (PER) (primary) — range: percent
  • Calculated as the normalized Levenshtein distance between the predicted and ground-truth sequences of punctuation tokens (periods, commas, question marks), isolating punctuation-specific errors.

Input / output format

Input: Audio recordings paired with reference transcripts preprocessed to retain only periods, commas, and question marks, with words and punctuation separated by single spaces.

Output: Predicted text sequence with words and punctuation/capitalization tokens separated by spaces.

Scoring recipe

def compute_per(pred_text, gold_text):
    # Extract only punctuation tokens as per benchmark preprocessing
    pred_punct = [w for w in pred_text.split() if w in {'.', ',', '?'}]
    gold_punct = [w for w in gold_text.split() if w in {'.', ',', '?'}]
    # Compute Levenshtein distance
    dist = levenshtein_distance(pred_punct, gold_punct)
    # Normalize by gold length to get error rate
    return dist / len(gold_punct) if len(gold_punct) > 0 else 0.0

Common pitfalls

• The benchmark restricts evaluation to only periods, commas, and question marks, ignoring other punctuation marks present in standard ASR datasets.
• Models must be evaluated on space-separated token sequences (e.g., 'done .') rather than raw concatenated text to match the preprocessing protocol.
• Comparing cascade and E2E models requires identical decoding settings (e.g., beam search with n-gram LMs) to avoid confounding factors.

Evidence (verbatim from paper)

It proposes a novel Punctuation Error Rate (PER) metric based on Levenshtein distance that isolates punctuation-specific errors, enabling fine-grained comparison between cascaded and end-to-end models.

Citation

@misc{meister2023librispeechpc,
  title={LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models},
  author={Meister et al. (2023)},
  year={2023},
  note={arXiv:2310.02943}
}

• arXiv: 2310.02943