Hyface Vc Eval

hyface-vc-eval

Hear Your Face: Face-based voice conversion with F0 estimation — Jaejun Lee et al. (2024) (arXiv:2408.09802, 2024)

What this evaluates

This protocol evaluates face-based voice conversion models by measuring how well synthesized audio matches the target speaker's identity and pitch characteristics using only facial images as input. It probes cross-modal alignment, speaker homogeneity, diversity, and explicit fundamental frequency (F0) estimation accuracy.

Datasets

• LRS3 — total 5502; splits: trainval (-1), test (-1)

Metrics

• Pitch deviation (primary) — range: Hz
  • Mean absolute difference between the frame-wise F0 of the synthesized audio and the average F0 of the ground-truth target speaker audio.
• Consistency(obj) — range: [-1, 1]
  • Cosine similarity between speaker embeddings of the synthesized audio and the ground-truth audio from the same target speaker.
• Homogeneity — range: [-1, 1]
  • Cosine similarity of speaker embeddings in synthesized audio generated from different facial images of the same speaker.
• Diversity — range: [-1, 1]
  • Cosine similarity of speaker embeddings in synthesized audio generated from different target speakers.
• Consistency(sub) — range: [1, 5]
  • 5-point Mean Opinion Score (MOS) assessing alignment between synthesized audio and corresponding facial images.
• Naturalness — range: [1, 5]
  • 5-point MOS assessing sound quality of synthesized audio.
• ABX test — range: percent
  • Subjective preference test where participants choose which of two synthesized audio samples better matches a given face image.

Input / output format

Input: Source audio clip and a set of frontal facial images (224×224, 25 fps) of the target speaker.

Output: Synthesized speech audio matching the target speaker's voice characteristics.

Scoring recipe

def compute_metrics(pred_audio, gold_audio):
    pred_f0 = extract_f0(pred_audio)
    gold_f0_avg = np.mean(extract_f0(gold_audio))
    pitch_dev = np.mean(np.abs(pred_f0 - gold_f0_avg))
    pred_emb = Resemblyzer.encode(pred_audio)
    gold_emb = Resemblyzer.encode(gold_audio)
    consistency_obj = cosine_similarity(pred_emb, gold_emb)
    return {'pitch_deviation': pitch_dev, 'consistency_obj': consistency_obj}

Common pitfalls

• Homogeneity and Diversity measure opposite goals (similarity within same speaker vs. distinctiveness across speakers); confusing them inverts the desired score direction.
• Consistency(rnd) is a control metric, not a performance target; high scores here indicate the model fails to capture the target speaker's identity.
• Ground-truth (GT) audio cannot be evaluated for heterogeneous gender pairings (HTG), so GT scores are only reported for homogeneous sets (HMG).

Evidence (verbatim from paper)

Following Sheng et al. [9] and other conventional VC studies, for objective evaluation, we assess the homogeneity, diversity, and objective consistency. For subjective evaluation, we examine subjective consistency, naturalness, and ABX tests. Furthermore, we propose a new evaluation metric: pitch deviation.

Citation

@misc{lee2024hearyourface,
  title={Hear Your Face: Face-based voice conversion with F0 estimation},
  author={Jaejun Lee et al. (2024)},
  year={2024},
  note={arXiv:2408.09802}
}

• arXiv: 2408.09802