Libriquote Eval

libriquote-eval

LibriQuote: A Speech Dataset of Fictional Character Utterances for Expressive Zero-Shot Speech Synthesis — Michel et al. (2025) (arXiv:2509.04072, 2025)

What this evaluates

Probes the ability of zero-shot text-to-speech systems to generate expressive, character-specific utterances while preserving reference speaker timbre. It evaluates cross-sentence generation where a narration clip guides the synthesis of a fictional quotation, testing prosodic variability, emotional expressiveness, and speech intelligibility.

Datasets

• LibriQuote — total ?; splits: train (-1), test (-1); repo https://github.com/deezer/libriquote

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed using Whisper-large-v3 transcription against the target text. Lower is better.
• SIM-O — range: [-1, 1]
  • Cosine similarity between speaker embeddings extracted from a WavLM-large verification model for synthesized and ground-truth audio. Higher is better.
• MCD — range: other
  • Mel Cepstral Distortion measuring spectral envelope difference between synthesized and ground-truth speech. Lower is better.
• FPC — range: [-1, 1]
  • Pearson correlation coefficient between fundamental frequency (F0) contours of synthesized and ground-truth speech. Higher is better.
• MOS — range: [1, 5]
  • Mean Opinion Score rated by human listeners on a scale from 1 to 5 (0.5 intervals) to assess naturalness.
• CMOS — range: [-3, 3]
  • Comparative MOS rated from -3 to 3, measuring the degree of expressivity of synthesized speech relative to ground-truth.

Input / output format

Input: Reference narration audio clip (2–15 seconds), target quotation text, and optionally left/right context text tokens.

Output: Synthesized audio waveform matching the target quotation text.

Scoring recipe

# Objective metrics
wer = whisper_transcribe(prediction_audio) / len(gold_text_words)
sim_o = cosine_similarity(wavlm_embed(prediction_audio), wavlm_embed(gold_audio))
mcd = compute_mel_cepstral_distortion(prediction_audio, gold_audio)
fpc = pearson_correlation(f0_contour(prediction_audio), f0_contour(gold_audio))

# Subjective metrics (human evaluation)
mos = mean([rater.score(prediction_audio, scale=[1,5]) for rater in raters])
cmos = mean([rater.score(prediction_audio, reference=gold_audio, scale=[-3,3]) for rater in raters])

Common pitfalls

• High WER on ground-truth quotations is expected due to high expressiveness and fictional delivery, not a data quality issue.
• Speaker similarity (SIM-O) is computed against the original ground-truth speaker, not the reference narration speaker used for zero-shot prompting, which can cause confusion.
• Subjective evaluation relies on a small sample size (30 utterances, 2 per speaker) with only 5 raters, limiting statistical power for CMOS/MOS.

Evidence (verbatim from paper)

For objective metrics, we report Word Error Rate (WER) computed with Whisper-large-v3 (Radford et al. [2023]) to measure speech intelligibility and speaker similarity between the synthesized speech and the original ground-truth speech (SIM-O). We employ a WavLM-large based speaker verification model (Chen et al. [2022]) to extract speaker embeddings and calculate cosine similarities. We evaluate prosody similarity with the ground truth quotation using Mel Cepstral Distortion (MCD) and F0 Pearson correlation (FPC) following (Huang et al. [2023]).

Citation

@misc{michel2025libriquote,
  title={LibriQuote: A Speech Dataset of Fictional Character Utterances for Expressive Zero-Shot Speech Synthesis},
  author={Michel et al. (2025)},
  year={2025},
  note={arXiv:2509.04072}
}

• arXiv: 2509.04072