V2c Animation Eval

v2c-animation-eval

V2C: Visual Voice Cloning — Chen et al. (2021) (arXiv:2111.12890, 2021)

What this evaluates

Evaluates visually-driven voice cloning by measuring speech quality, temporal alignment, length consistency, speaker identity preservation, and emotion transfer accuracy against ground-truth audio.

Datasets

• V2C-Animation — total 10217; splits: test (-1)

Metrics

• MCD-DTW-SL (primary) — range: other
  • MCD-DTW weighted by Speech Length. Computes DTW minimum cumulative distance gamma_M,N between generated and ground-truth MFCC sequences, then scales by eta/R where eta = max(M,N)/min(M,N) and R is the number of DTW steps.
• MCD — range: other
  • Mel Cepstral Distortion. Sums Euclidean distance over first K MFCC values across T frames: (1/T) * sum(sqrt(sum((c_tk - c'_tk)^2))).
• Id. Acc. — range: percent
  • Speaker identity accuracy. Computes cosine similarity between generated speech embedding and pre-computed speaker centroids (from GE2E encoder), then calculates percentage of correct classifications.
• Emo. Acc. — range: percent
  • Emotion accuracy. Same procedure as Id. Acc. but uses emotion centroids derived from reference video labels.
• MOS-naturalness — range: [1, 5]
  • Mean Opinion Score for naturalness. Listeners rate generated audio on an ACR scale from 1 (Bad) to 5 (Excellent) in 0.5 increments.
• MOS-similarity — range: [1, 5]
  • Mean Opinion Score for similarity. Listeners rate how well generated speech aligns with desired voice/prosody compared to ground truth on a 1-5 ACR scale.

Input / output format

Input: Reference audio clip, reference video clip, and target text transcript.

Output: Synthesized speech audio waveform.

Scoring recipe

def compute_mcd_dtw_sl(c, c_gt):
    gamma = dtw_minimum_distance(c, c_gt)
    R = max(len(c), len(c_gt))
    eta = max(len(c), len(c_gt)) / min(len(c), len(c_gt))
    return (eta / R) * gamma

def compute_accuracy(audio, gold_label, centroids):
    emb = ge2e_encoder(audio)
    pred = argmax([cosine_similarity(emb, cent) for cent in centroids])
    return (pred == gold_label)

# Aggregate over test set
mcd_dtw_sl_scores = [compute_mcd_dtw_sl(gen, gt) for gen, gt in test_set]
acc_scores = [compute_accuracy(gen, label, centroids) for gen, label in test_set]

Common pitfalls

• Standard MCD requires equal-length inputs; zero-padding shorter speech can artificially inflate distortion if mismatches occur early in the sequence.
• Plain MCD-DTW ignores length mismatch, allowing artificially low scores if any segment aligns, which is why the length-weighted variant (MCD-DTW-SL) is necessary.
• Identity and emotion accuracy scores are capped below 100% even on ground truth due to imperfections in the pre-trained GE2E classifier and emotion encoder.

Evidence (verbatim from paper)

To assess the quality of generated speech, we use Mel Cepstral Distortion (MCD) metric... To alleviate the above issues, we propose a MCD-DTW weighted by Speech Length (MCD-DTW-SL)... we use a Mean Opinion Score (MOS) evaluation approach... To evaluate whether the generated speech carries proper speaker identity and emotion, we propose an identity accuracy and an emotion accuracy, respectively.

Citation

@misc{chen2021v2c,
  title={V2C: Visual Voice Cloning},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2111.12890}
}

• arXiv: 2111.12890