Cross Lingual F5 Tts Eval

cross-lingual-f5-tts-eval

Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis — Liu et al. (2025) (arXiv:2509.14579, 2025)

What this evaluates

Evaluates the intelligibility, speaker similarity, and naturalness of synthesized speech in cross-lingual voice cloning and TTS scenarios. It also measures the accuracy of a language-agnostic speaking rate predictor for duration modeling across multiple languages.

Datasets

• Emilia — total ?; splits: train (-1)
• Seed-TTS-eval — total ?; splits: test-en (-1), test-zh (-1)
• LibriSpeech-PC test-clean — total ?; splits: test-clean (-1)
• FLEURS — total 473; splits: test (473)

Metrics

• WER (primary) — range: percent
  • Measures intelligibility by comparing the automatic transcription of synthesized speech with the ground truth text using Whisper-large-V3 or Paraformer-zh.
• SIM-o — range: [0, 1]
  • Quantifies speaker resemblance by extracting embeddings with a WavLM-large-based verification model and computing the cosine similarity between reference and synthesized audio.
• UTMOS — range: other
  • Provides an automated assessment of speech naturalness using a pre-trained MOS prediction model that estimates audio quality without reference recordings.
• MRE — range: percent
  • Measures relative duration prediction accuracy as the average relative difference between predicted duration (linguistic units divided by predicted speaking rate) and ground truth audio duration.
• MAE — range: other
  • Quantifies absolute duration prediction deviation as the mean absolute difference between predicted and ground truth audio duration.

Input / output format

Input: Audio prompt (3–8 seconds), target text, and reference audio (for speaker similarity evaluation).

Output: Synthesized speech audio waveform.

Scoring recipe

# WER
pred_text = whisper_large_v3.synthesize(synthesized_audio)
wer = compute_wer(pred_text, ground_truth_text)

# SIM-o
ref_emb = wavlm_large(reference_audio)
syn_emb = wavlm_large(synthesized_audio)
sim_o = cosine_similarity(ref_emb, syn_emb)

# UTMOS
utmos_score = utmos_model.predict(synthesized_audio)

# MAE & MRE (speaking rate predictor)
pred_duration = num_linguistic_units / predicted_speaking_rate
mae = mean(abs(pred_duration - ground_truth_duration))
mre = mean(abs(pred_duration - ground_truth_duration) / ground_truth_duration)

Common pitfalls

• Using Whisper-X transcriptions directly without the paper's specialized preprocessing to skip anomalous tokens (digits, symbols, other languages) causes forced alignment failures.
• Confusing the speaking rate predictor's evaluation metrics (MAE/MRE on duration) with the main TTS generation metrics (WER/SIM-o/UTMOS).
• Not adhering to the specified inference hyperparameters (Euler ODE solver, NFE=32, CFG=2.0, sway sampling=-1.0, Vocos vocoder), which significantly impact synthesis quality.

Evidence (verbatim from paper)

We follow the evaluation setting of F5-TTS, adopting Seed-TTS-eval and LibriSpeech-PC test-clean*[[20]]* as our test set. We also build a multilingual cross-lingual test set with 473 samples of 3-8 second audio prompts from FLEURS*[[21]], covering four languages (German, French, Hindi, Korean) to synthesize both English and Chinese speech. The evaluation is conducted using the following three metrics: Word Error Rate (WER) measures the intelligibility of synthesized speech by comparing its transcription with the ground truth text. We employ Whisper-large-V3[[22]]* and Paraformer-zh*[[23]]* for automatic recognition and compute WER accordingly. Speaker Similarity (SIM-o) quantifies the resemblance between the synthesized and the original target speeches. We use WavLM-large-based*[[24]]* speaker verification model to extract speaker embeddings and compute the cosine similarity between them. UTMOS*[[25]]* provides an automated assessment of speech naturalness through a pre-trained MOS prediction model.

Citation

@misc{liu2025crosslingualf5tts,
  title={Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2509.14579}
}

• arXiv: 2509.14579