Librispeech Asr Correction Eval

librispeech-asr-correction-eval

Towards interfacing large language models with ASR systems using confidence measures and prompting — Naderi et al. (2024) (arXiv:2407.21414, 2024)

What this evaluates

Evaluates confidence-based filtering strategies for applying LLMs to post-hoc correction of ASR transcripts, measuring how well the system reduces transcription errors in low-confidence segments while preserving accurate outputs.

Datasets

• LibriSpeech — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1); HF librispeech_asr

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions required to transform the predicted transcript into the reference transcript, divided by the number of words in the reference transcript.

Input / output format

Input: ASR transcript with associated sentence- and word-level confidence scores, processed through a confidence threshold filter before being passed to an LLM prompt for correction.

Output: Corrected ASR transcript (or the original unmodified transcript if confidence scores exceed the filtering threshold).

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_ref_words = 0
    for pred, ref in zip(predictions, references):
        # Standard Levenshtein-based edit distance
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_ref_words += len(ref.split())
    return (total_errors / total_ref_words) * 100 if total_ref_words > 0 else 0.0

Common pitfalls

• LLMs may introduce new errors into high-confidence transcripts if filtering thresholds are set too loosely.
• Whisper's confidence scores are averaged token log-probabilities, which may not perfectly correlate with actual word-level errors.
• Potential data contamination risk as the proprietary ChatGPT models may have been trained on LibriSpeech data.

Evidence (verbatim from paper)

We evaluate our proposed approach on the English LibriSpeech corpus [27] of audiobook recordings. We use the dev-clean and dev-other subsets for initial experiments and hyperparameter tuning and then report final results on the test-clean and test-other evaluation sets. Each of these subsets contains around 2500-3000 utterances. Speakers in the other portions are more challenging to recognize and lead to higher WERs.

Citation

@misc{naderi2024asrllm,
  title={Towards interfacing large language models with ASR systems using confidence measures and prompting},
  author={Naderi et al. (2024)},
  year={2024},
  note={arXiv:2407.21414}
}

• arXiv: 2407.21414