Lip2wav Eval

lip2wav-eval

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis — Prajwal et al. (2020) (arXiv:2005.08209, 2020)

What this evaluates

Evaluates a model's ability to synthesize natural, speaker-specific speech from unconstrained lip movements in large-vocabulary settings. It measures how well the model captures individual speaking styles and contextual cues from video frames.

Datasets

• Lip2Wav — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/Rudrabha/Lip2Wav
• GRID — total ?; splits: train (-1), val (-1), test (-1)
• TCD-TIMIT lip speaker corpus — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• mel reconstruction loss (primary) — range: other
  • Computed as the mean squared error (MSE) between the predicted and ground-truth mel-spectrogram frames: L = (1/N) Σ (y_pred - y_true)². Lower values indicate better spectral reconstruction fidelity.

Input / output format

Input: Contiguous 3-second sequence of video frames (face crops resized to 48×48) representing lip movements.

Output: Generated speech waveform (produced via Griffin-Lim algorithm from predicted mel-spectrograms).

Scoring recipe

def compute_mel_reconstruction_loss(pred_mels, gt_mels):
    # pred_mels and gt_mels: tensors of shape (T, F)
    loss = torch.mean((pred_mels - gt_mels) ** 2)
    return loss.item()

Common pitfalls

• Video-level splitting is mandatory to prevent data leakage between train and test sets.
• Griffin-Lim vocoder is explicitly used over neural vocoders due to lower accuracy of generated mel-spectrograms.
• Sliding window inference requires overlap to adjust for boundary effects, which must be accounted for during evaluation.

Evidence (verbatim from paper)

train until the mel reconstruction loss plateaus for at least 30K iterations. ... The model with the best performance on the validation set is chosen for testing and evaluation.

Citation

@misc{prajwal2020lip2wav,
  title={Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis},
  author={Prajwal et al. (2020)},
  year={2020},
  note={arXiv:2005.08209}
}

• arXiv: 2005.08209