Voiceloop Tts Eval

voiceloop-tts-eval

VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop — Taigman et al. (2017) (arXiv:1707.06588, 2017)

What this evaluates

Evaluates a text-to-speech model's ability to synthesize perceptually natural speech and accurately mimic speaker identities from text and reference embeddings. It measures robustness across clean benchmarks, multi-speaker corpora, and noisy in-the-wild recordings, while testing few-shot voice fitting capabilities.

Datasets

• LJ (LJSpeech) — total ?; splits: (unstated)
• Nancy (Blizzard 2011) — total ?; splits: (unstated)
• Blizzard 2013 Audiobook — total ?; splits: (unstated)
• VCTK — total ?; splits: train (-1), test (-1), val (8)
• In-the-wild YouTube speeches — total ?; splits: (unstated)

Metrics

• MOS (primary) — range: [1, 5]
  • Average of human listener ratings on a 1-5 scale for speech naturalness and quality, computed via the crowdMOS toolkit.
• MCD DTW — range: dB
  • Mel Cepstral Distortion computed along the optimal dynamic time warping path between predicted and reference mel-cepstral sequences to handle misalignment.
• Speaker Identification Accuracy — range: [0, 1]
  • Top-1 classification accuracy of a CNN trained on ground-truth audio, tested on generated audio to measure voice identity preservation.

Input / output format

Input: Text transcript (character-level) and a speaker embedding vector. For fitting experiments, a short reference audio segment is used to initialize or update the embedding.

Output: Synthesized audio waveform sampled at 16 kHz.

Scoring recipe

def compute_mcd_dtw(pred_wav, gold_wav, sr=16000):
    pred_mcf = extract_mel_cepstrum(pred_wav, sr)
    gold_mcf = extract_mel_cepstrum(gold_wav, sr)
    dist_matrix = cdist(pred_mcf, gold_mcf, metric='euclidean')
    path = dtw_path(dist_matrix)
    mcd = (10.0 / np.log(10)) * np.sqrt(2 * np.sum(dist_matrix[path])) / len(path)
    return mcd

Common pitfalls

• MOS scores are highly dependent on rater demographics and listening conditions; the paper restricts raters to North Americans, limiting cross-cultural generalizability.
• MCD DTW assumes mel-cepstral features are comparable; unaligned sequences can yield misleading distortion scores if DTW fails to find a monotonic path.
• Speaker identification accuracy relies on a separate classifier trained on ground-truth data, which may not reflect real-world deployment conditions or generalization to unseen speakers.

Evidence (verbatim from paper)

The MOS measure for the proposed method was computed using the crowdMOS toolkit by P. Ribeiro et al. (2011) and Amazon Mechanical Turk. All samples were presented at 16kHz and the raters were told that they are presented with the results of the different algorithms. At least 20 raters participated in each such experiment, with 95% confidence intervals. We restricted all experiments to North American raters. Tab. 3 presents Mel Cepstral Distortion (MCD) scores. This is an automatic, albeit limited, method of testing compatibility between two audio sequences. Since the sequences are not aligned, we employ MCD DTW, which uses dynamic time warping (DTW) to align the sequences.

Citation

@misc{taigman2017voiceloop,
  title={VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop},
  author={Taigman et al. (2017)},
  year={2017},
  note={arXiv:1707.06588}
}

• arXiv: 1707.06588