Unispeaker Mvc Eval

unispeaker-mvc-eval

Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation — Sheng et al. (2025) (arXiv:2501.06394, 2025)

What this evaluates

Evaluates multimodal voice generation and conversion capabilities across face-driven, text-driven, and attribute-based tasks. Probes the model's ability to align facial, textual, and attribute descriptions with target speech while preserving speaker identity, content clarity, and naturalness.

Datasets

• LRS3 — total ?; splits: train (-1), test (-1)

Metrics

• MOS-Match (primary) — range: other
  • Mean Opinion Score for voice suitability, measuring how well the generated voice matches the target modality (face, text, or attributes). Evaluated via human listening tests on a standard scale.
• MOS-Nat — range: other
  • Mean Opinion Score for naturalness, assessing the perceptual naturalness of the generated speech without reference to the target modality.
• SST — range: other
  • Speaker Similarity Score, quantifying the acoustic similarity between the generated voice and the target speaker's identity, typically computed via cosine similarity of speaker embeddings.
• SSD — range: other
  • Speech Distortion Score, measuring the level of acoustic distortion or noise introduced during generation, often using PESQ or STOI.
• SSC — range: other
  • Speech Clarity Score, evaluating the intelligibility and overall quality of the generated speech, typically assessed via human listening tests.

Input / output format

Input: Multimodal conditioning inputs (face images, textual descriptions, or voice attributes) combined with content text for TTS or reference speech for VC.

Output: Generated audio waveform/speech.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: generated audio files
    # gold: target audio / modality descriptions
    mos_match = human_listen_test(predictions, gold['modality'], aspect='suitability')
    mos_nat = human_listen_test(predictions, aspect='naturalness')
    sst = cosine_similarity(speaker_embeddings(predictions), speaker_embeddings(gold['target_speaker']))
    ssd = pesq_or_stoi(predictions, gold['target_speaker'])
    ssc = human_listen_test(predictions, aspect='clarity')
    return {'MOS-Match': mos_match, 'MOS-Nat': mos_nat, 'SST': sst, 'SSD': ssd, 'SSC': ssc}

Common pitfalls

• MOS-Match and MOS-Nat are distinct subjective metrics; confusing them leads to incorrect suitability vs. quality reporting.
• ABX test results are highly dependent on listener demographics and audio playback conditions, requiring strict standardization.
• Performance scales non-linearly with multimodal dataset size; evaluating on small subsets without reporting scale may misrepresent capability.

Evidence (verbatim from paper)

Across five tasks, UniSpeaker outperformed previous approaches on all three metrics, except for MOS-Match in the AVE task.

Citation

@misc{sheng2025unispeaker,
  title={Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation},
  author={Sheng et al. (2025)},
  year={2025},
  note={arXiv:2501.06394}
}

• arXiv: 2501.06394