Talkbank Asr Eval

talkbank-asr-eval

ASR Benchmarking: Need for a More Representative Conversational Dataset — Maheshwari et al. (2024) (arXiv:2409.12042, 2024)

What this evaluates

Evaluates the robustness of state-of-the-art automatic speech recognition (ASR) models on real-world, unstructured conversational speech compared to controlled, read-speech benchmarks. It specifically probes how conversational disfluencies, interruptions, and variable audio durations impact transcription accuracy.

Datasets

• TalkBank — total ?; splits: test (-1); repo https://github.com/Diabolocom-Research/ConversationalDataset

Metrics

• Word Error Rate (primary) — range: [0, 1]
  • Normalized Word Error Rate calculated as (Insertions + Deletions + Substitutions) / Total Reference Words. Reported as a decimal between 0 and 1.
• Pearson correlation — range: [-1, 1]
  • Measures the linear correlation between the normalized count of conversational-specific markers (e.g., pauses, laughter, interruptions) and the Word Error Rate per transcript.

Input / output format

Input: Raw audio files (wav) of varying durations, including short segments (<20s) and long continuous speaker-switch recordings with silences.

Output: Transcribed text string corresponding to the audio input.

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    if total_words == 0: return 0.0
    edits = 0
    for pred, ref in zip(predictions, references):
        # Standard Levenshtein edit distance on word tokens
        edits += levenshtein_distance(pred.split(), ref.split())
    return edits / total_words

Common pitfalls

• TalkBank Segments and TalkBank Switch are distinct variants with different audio lengths and silence patterns; results are not directly comparable without noting this distinction.
• Word Error Rate is reported as a decimal (e.g., 0.22) rather than a percentage, which can cause confusion when comparing to other ASR benchmarks.
• wav2vec2 was excluded from the Speaker Switch variant due to memory constraints on long audio files, so its results are only available for the Segments variant.

Evidence (verbatim from paper)

In all these datasets, we rely on the test sets to evaluate the zero-shot performance of ASR models. We report the normalized Word Error Rate for all experiments.

Citation

@misc{maheshwari2024asrbenchmarking,
  title={ASR Benchmarking: Need for a More Representative Conversational Dataset},
  author={Maheshwari et al. (2024)},
  year={2024},
  note={arXiv:2409.12042}
}

• arXiv: 2409.12042