Styledubber Eval

styledubber-eval

StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing — Cong et al. (2024) (arXiv:2402.12636, 2024)

What this evaluates

Evaluates a visual-to-audio dubbing model's ability to generate emotionally consistent, speaker-identical speech that aligns temporally with video lip movements. It probes multi-scale style learning across unseen speakers, reference audio variations, and phoneme-level lip-sync accuracy.

Datasets

• V2C-Animation — total 10217; splits: test (-1)
• GRID — total 33000; splits: train (32670), test (3300)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed by transcribing the generated audio with Whisper and comparing it to the ground truth script text. Lower values indicate better pronunciation accuracy.
• SPK-SIM — range: [0, 1]
  • Speaker Identity Similarity score, typically calculated as the cosine similarity between speaker embeddings extracted from a pre-trained speaker verification model for generated and reference audio.
• EMO-ACC — range: [0, 1]
  • Emotion Accuracy, measured by the classification accuracy of an emotion recognizer on the generated audio compared to the ground-truth emotion labels.
• MCD-DTW — range: other
  • Mel Cepstral Distortion computed using Dynamic Time Warping to align generated and reference mel-spectrograms, measuring acoustic difference independent of duration.
• MCD-DTW-SL — range: other
  • MCD-DTW weighted by duration consistency between the generated and reference phoneme sequences, penalizing temporal misalignment.
• MOS-N — range: [1, 5]
  • Mean Opinion Score for Naturalness, a 5-point subjective rating from human listeners assessing the realism of the generated speech.
• MOS-S — range: [1, 5]
  • Mean Opinion Score for Similarity, a 5-point subjective rating from human listeners assessing how closely the generated speech matches the reference speaker and emotion.

Input / output format

Input: Video frames sampled at 25 FPS (lip region resized to 96x96), reference audio, target script text, and emotion annotations.

Output: Time-domain audio waveforms (22.05kHz) generated from 80-channel mel-spectrograms via a HiFi-GAN vocoder.

Scoring recipe

def evaluate(generated_audio, gold_audio, gold_text):
    # WER
    asr_text = whisper.transcribe(generated_audio, language='en')['text']
    wer = wer_score(gold_text, asr_text) * 100
    # MCD-DTW
    gen_mel = extract_melspectrogram(generated_audio)
    ref_mel = extract_melspectrogram(gold_audio)
    mcd_dtw = compute_mcd_with_dtw(gen_mel, ref_mel)
    # MCD-DTW-SL
    duration_weight = compute_duration_consistency(gold_text, asr_text)
    mcd_dtw_sl = mcd_dtw * duration_weight
    # SPK-SIM & EMO-ACC
    spk_sim = cosine_similarity(get_speaker_emb(generated_audio), get_speaker_emb(gold_audio))
    emo_acc = emotion_classifier.predict(generated_audio) == gold_emotion
    return {'WER': wer, 'MCD-DTW': mcd_dtw, 'MCD-DTW-SL': mcd_dtw_sl, 'SPK-SIM': spk_sim, 'EMO-ACC': emo_acc}

Common pitfalls

• Dub 1.0 setting uses ground-truth audio as reference, which is unrealistic for real-world dubbing tasks and inflates performance compared to Dub 2.0/3.0.
• MCD-DTW-SL differs from standard MCD-DTW by incorporating duration consistency weighting; comparing them directly without accounting for the weight leads to unfair benchmarking.
• Subjective MOS evaluations only sample 25 random audio clips per test set, which may not capture full dataset variability or speaker-specific nuances.

Evidence (verbatim from paper)

To measure whether the generated speech carries the desired speaker identity and emotion, speaker identity similarity (SPK-SIM) and emotion accuracy (EMO-ACC) are employed as in Chen et al. (2022). We adopt the Mel Cepstral Distortion Dynamic Time Warping (MCD-DTW) to measure the difference between generated speech and real speech. We also adopt the metric MCD-DTW-SL, which is MCD-DTW weighted by duration consistency Chen et al. (2022). Besides, the Word Error Rate (WER) is used to measure pronunciation accuracy by the publicly available whisper Radford et al. (2023) as the ASR model.

Citation

@misc{cong2024styledubber,
  title={StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing},
  author={Cong et al. (2024)},
  year={2024},
  note={arXiv:2402.12636}
}

• arXiv: 2402.12636