Cloze Ger Eval

cloze-ger-eval

Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models — Hu et al. (2024) (arXiv:2405.10025, 2024)

What this evaluates

Evaluates a model's ability to perform generative error correction (GER) for automatic speech recognition by reformulating the task as a cloze test. The model must select the correct hypothesis from a 5-best N-best list to minimize word error rate while maintaining source speech fidelity.

Datasets

• HyPoradise (GER benchmark) — total ?; splits: test (-1)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate (WER) is calculated as the percentage of substitutions, deletions, and insertions required to transform the predicted hypothesis into the ground truth reference, relative to the total number of words in the reference. WER = (S + D + I) / N × 100.

Input / output format

Input: Cloze test prompt with a masked token and 5 candidate options (A–E) derived from Whisper-Large N-best hypotheses. Optionally includes source speech audio features processed via SpeechGPT.

Output: Model selects one of the 5 options (A, B, C, D, or E) or generates a corrected text sequence. The final output is the selected/corrected hypothesis string.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Standard edit distance based WER calculation
        s, d, i = wer(ref, pred)
        total_errors += s + d + i
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Selection bias heavily favors option 'A' due to imbalanced training label distribution, requiring logits calibration during inference.
• N-best hypothesis quality varies across domains, causing context errors that necessitate a post-processing stage for optimal WER.
• Evaluating without source speech input significantly degrades performance compared to the SpeechGPT-augmented setup.

Evidence (verbatim from paper)

We utilize the HyPoradise (HP) dataset from the original GER benchmark*(Chen et al., [2023b])* for our experiments, which contains over 332K hypotheses-transcription pairs collected from multiple mainstream ASR corpora. Specifically, each transcription is paired with 5-best hypotheses transcribed from Whisper-Large model*(Radford et al., [2023])* with beam search decoding. In this work, we select 9 popular ASR corpora from HyPoradise to evaluate the proposed ClozeGER, including WSJ, CommonVoice, TED-LIUM3, SwitchBoard, LibriSpeech, CHiME-4, LRS2, ATIS, and CORAAL. Table 1: WER (%) results of ClozeGER with SpeechGPT and LoRA.

Citation

@misc{hu2024listenagain,
  title={Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models},
  author={Hu et al. (2024)},
  year={2024},
  note={arXiv:2405.10025}
}

• arXiv: 2405.10025