Lrs3 Avger Eval

lrs3-avger-eval

Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition — Liu et al. (2025) (arXiv:2501.04038, 2025)

What this evaluates

This evaluation probes the ability of a generative error correction model to refine audio-visual speech recognition transcripts under varying noise conditions. It measures how effectively multimodal cues (lip video and audio) combined with N-best hypotheses can reduce transcription errors compared to baseline systems.

Datasets

• LRS3 — total 151819; splits: train (118516), val (31982), test (1321)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the minimum number of word edits (insertions, deletions, substitutions) required to change the predicted transcript into the reference transcript, divided by the total number of words in the reference. Reported as a percentage.
• WERR — range: percent
  • Word Error Rate Reduction: calculated as (WER_baseline - WER_model) / WER_baseline * 100, measuring the relative improvement over the GER baseline.

Input / output format

Input: Per instance: a video clip (cropped 96x96 mouth ROI), corresponding audio, and a cross-modal prompt containing the 10-best ASR hypotheses and multimodal compression representations extracted via AV-HuBERT and a Q-Former.

Output: A single corrected transcript string generated by the fine-tuned LLaMA-7B model.

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    errors = sum(edit_distance(pred.split(), ref.split()) for pred, ref in zip(predictions, references))
    return (errors / total_words) * 100 if total_words > 0 else 0.0

def compute_werr(wer_model, wer_baseline):
    return ((wer_baseline - wer_model) / wer_baseline) * 100 if wer_baseline > 0 else 0.0

Common pitfalls

• WER is highly sensitive to the quality of the N-best hypotheses provided as input; poor initial ASR outputs limit correction potential.
• The paper re-splits LRS3 for training due to compute constraints, so direct comparison with original LRS3 benchmarks requires caution.
• Performance varies non-monotonically with SNR; models may rely more on lip cues at low SNR and audio/text at high SNR, affecting generalization expectations.

Evidence (verbatim from paper)

We evaluate performance using WER (Davis and Mermelstein [1980]), and Word Error Rate Reduction (WERR) (Leng et al. [2021]). Lower values of WER indicate better performance, while a higher WERR signifies a greater improvement relative to the GER approach.

Citation

@misc{liu2025listening,
  title={Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2501.04038}
}

• arXiv: 2501.04038