Lip To Speech Eval

lip-to-speech-eval

SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis — Liang et al. (2026) (arXiv:2602.11477, 2026)

What this evaluates

Evaluates a model's ability to synthesize high-fidelity, intelligible speech directly from visual lip movements. It probes perceptual audio quality, content accuracy, and speaker identity preservation in a cross-dataset generalization setting.

Datasets

• LRS3-TED — total 150000; splits: train (-1), val (-1), test (-1)
• LRS2-BBC — total 144000; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated using the AUTO-AVSR model, measuring the percentage of incorrectly recognized words relative to the ground truth transcript.
• UTMOS — range: [1, 5]
  • Non-intrusive reference-free model predicting a mean opinion score (MOS) on a 1-to-5 scale to correlate with human quality judgments.
• SCOREQ — range: [1, 5]
  • Reference-free model predicting a MOS on a 1-to-5 scale for perceptual quality assessment.
• D-BERT — range: [-1, 1]
  • Cosine similarity between SSL embeddings of synthesized and target speech, measuring semantic similarity.
• SECS — range: [-1, 1]
  • Cosine similarity between speaker embeddings extracted from synthesized and target speech using Resemblyzer, measuring speaker identity preservation.
• MOS — range: [1, 5]
  • Human-rated Mean Opinion Score on a 1-to-5 scale assessing naturalness, intelligibility, and speaker similarity.

Input / output format

Input: Mouth region-of-interest video frames (88x88 grayscale) processed into 1024-dimensional visual features via AV-Hubert Large, plus a 256-dimensional speaker identity embedding from a reference utterance.

Output: Continuous latent vectors of the X-Codec-hubert neural audio codec, which are subsequently decoded into 16 kHz audio waveforms.

Scoring recipe

def compute_metrics(synthesized_audio, target_audio, target_transcript):
    wer = auto_avsr.transcribe(synthesized_audio) / len(target_transcript)
    d_bert = cosine_similarity(hubert_embed(target_audio), hubert_embed(synthesized_audio))
    secs = cosine_similarity(resemblyzer_embed(target_audio), resemblizer_embed(synthesized_audio))
    utmos = utmos_model.predict(synthesized_audio)
    scoreq = scoreq_model.predict(synthesized_audio)
    return {'WER': wer, 'D-BERT': d_bert, 'SECS': secs, 'UTMOS': utmos, 'SCOREQ': scoreq}

Common pitfalls

• Using standard ASR instead of AUTO-AVSR for WER calculation, which ignores visual cues and inflates error rates.
• Confusing reference-free metrics (UTMOS, SCOREQ) with intrusive metrics; these require no target audio and predict MOS directly.
• Subjective MOS evaluation requires exactly 15 participants rating 30 samples per criterion; deviating from this protocol invalidates comparability.

Evidence (verbatim from paper)

We perform a comprehensive evaluation of our proposed SLD-L2S framework using a suite of objective and subjective metrics. The evaluation is designed to assess three critical aspects of the synthesized speech: perceptual quality, content intelligibility, and speaker similarity. For objective evaluation, we employ the following metrics: Quality: We assess perceptual quality using UTMOS and SCOREQ. Both are non-intrusive, reference-free models that predict a mean opinion score (MOS) on a 1-to-5 scale, designed to correlate with human quality judgments. Intelligibility: We measure intelligibility from two perspectives. First, the word error rate (WER) is calculated using the AUTO-AVSR. Second, we utilize SpeechBERTScore (D-BERT)... Speaker Similarity: We quantify speaker identity preservation using speaker embedding cosine similarity (SECS).

Citation

@misc{liang2026sldl2s,
  title={SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis},
  author={Liang et al. (2026)},
  year={2026},
  note={arXiv:2602.11477}
}

• arXiv: 2602.11477