# Hyface Vc Eval

> This protocol evaluates face-based voice conversion models by measuring how well synthesized audio matches the target speaker's identity and pitch characteristics using only facial images as input. It probes cross-modal alignment, speaker homogeneity, diversity, and explicit fundamental frequency (F0) estimation accuracy. Use when the user wants to benchmark on LRS3, or asks about evaluating this task. Reports Pitch deviation.

- Skill: `qhjqhj00/hyface-vc-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/hyface-vc-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/hyface-vc-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/hyface-vc-eval

---


# hyface-vc-eval

> Hear Your Face: Face-based voice conversion with F0 estimation — Jaejun Lee et al. (2024) (arXiv:2408.09802, 2024)

## What this evaluates

This protocol evaluates face-based voice conversion models by measuring how well synthesized audio matches the target speaker's identity and pitch characteristics using only facial images as input. It probes cross-modal alignment, speaker homogeneity, diversity, and explicit fundamental frequency (F0) estimation accuracy.

## Datasets

- **LRS3** — total 5502; splits: trainval (-1), test (-1)

## Metrics

- `Pitch deviation` **(primary)** — range: Hz
  - Mean absolute difference between the frame-wise F0 of the synthesized audio and the average F0 of the ground-truth target speaker audio.
- `Consistency(obj)` — range: [-1, 1]
  - Cosine similarity between speaker embeddings of the synthesized audio and the ground-truth audio from the same target speaker.
- `Homogeneity` — range: [-1, 1]
  - Cosine similarity of speaker embeddings in synthesized audio generated from different facial images of the same speaker.
- `Diversity` — range: [-1, 1]
  - Cosine similarity of speaker embeddings in synthesized audio generated from different target speakers.
- `Consistency(sub)` — range: [1, 5]
  - 5-point Mean Opinion Score (MOS) assessing alignment between synthesized audio and corresponding facial images.
- `Naturalness` — range: [1, 5]
  - 5-point MOS assessing sound quality of synthesized audio.
- `ABX test` — range: percent
  - Subjective preference test where participants choose which of two synthesized audio samples better matches a given face image.

## Input / output format

**Input**: Source audio clip and a set of frontal facial images (224×224, 25 fps) of the target speaker.

**Output**: Synthesized speech audio matching the target speaker's voice characteristics.

## Scoring recipe

```python
def compute_metrics(pred_audio, gold_audio):
    pred_f0 = extract_f0(pred_audio)
    gold_f0_avg = np.mean(extract_f0(gold_audio))
    pitch_dev = np.mean(np.abs(pred_f0 - gold_f0_avg))
    pred_emb = Resemblyzer.encode(pred_audio)
    gold_emb = Resemblyzer.encode(gold_audio)
    consistency_obj = cosine_similarity(pred_emb, gold_emb)
    return {'pitch_deviation': pitch_dev, 'consistency_obj': consistency_obj}
```

## Common pitfalls

- Homogeneity and Diversity measure opposite goals (similarity within same speaker vs. distinctiveness across speakers); confusing them inverts the desired score direction.
- Consistency(rnd) is a control metric, not a performance target; high scores here indicate the model fails to capture the target speaker's identity.
- Ground-truth (GT) audio cannot be evaluated for heterogeneous gender pairings (HTG), so GT scores are only reported for homogeneous sets (HMG).

## Evidence (verbatim from paper)

> Following Sheng et al. [9] and other conventional VC studies, for objective evaluation, we assess the homogeneity, diversity, and objective consistency. For subjective evaluation, we examine subjective consistency, naturalness, and ABX tests. Furthermore, we propose a new evaluation metric: pitch deviation.

## Citation

```bibtex
@misc{lee2024hearyourface,
  title={Hear Your Face: Face-based voice conversion with F0 estimation},
  author={Jaejun Lee et al. (2024)},
  year={2024},
  note={arXiv:2408.09802}
}
```

- arXiv: 2408.09802