Libris2s Tts Eval

libris2s-tts-eval

LibriS2S: A German-English Speech-to-Speech Translation Corpus — Jeuris et al. (2022) (arXiv:2204.10593, 2022)

What this evaluates

Evaluates the acoustic quality and prosodic fidelity of generated German-English speech-to-speech translation audio. It measures perceived naturalness via an automated MOS approximation, and quantifies pitch and energy accuracy against ground truth references.

Datasets

• LibriS2S (Frankenstein subset) — total 2335; splits: train (2079), val (129), test (127)

Metrics

• MOSNet score (primary) — range: [0, 5]
  • Approximates human Mean Opinion Score (MOS) using a pre-trained deep learning model (MOSNet) applied to the generated audio waveform.
• Pitch moments (std, skewness, kurtosis) — range: other
  • Statistical moments of the pitch contour distribution in the generated speech, calculated to match ground truth prosody.
• DTW distance (pitch) — range: other
  • Average Dynamic Time Warping distance between the pitch contour of the generated audio and the ground truth audio.
• Energy MAE — range: other
  • Mean Absolute Error between the energy of the generated speech and ground truth speech, computed over durations aligned by MFA phoneme timings.

Input / output format

Input: Aligned source audio (German) and target transcript (English), plus ground truth target audio (English) for reference during evaluation.

Output: Synthesized target audio (English) waveform generated by the TTS model.

Scoring recipe

def evaluate(generated_audio, ground_truth_audio, mfa_durations):
    # 1. MOSNet approximation
    mos_score = MOSNet.predict(generated_audio)
    
    # 2. Pitch metrics
    pitch_gen = extract_pitch(generated_audio)
    pitch_gt = extract_pitch(ground_truth_audio)
    pitch_std = np.std(pitch_gen)
    pitch_skew = skewness(pitch_gen)
    pitch_kurt = kurtosis(pitch_gen)
    dtw_dist = dtw(pitch_gen, pitch_gt)
    
    # 3. Energy MAE (strictly using MFA durations for alignment)
    energy_gen = extract_energy(generated_audio, mfa_durations)
    energy_gt = extract_energy(ground_truth_audio, mfa_durations)
    energy_mae = np.mean(np.abs(energy_gen - energy_gt))
    
    return {'mosnet': mos_score, 'pitch_std': pitch_std, 'pitch_skew': pitch_skew, 
            'pitch_kurt': pitch_kurt, 'dtw_dist': dtw_dist, 'energy_mae': energy_mae}

Common pitfalls

• MOSNet is an automated approximation of human MOS and may not perfectly correlate with real human perceptual judgments.
• Energy MAE calculation strictly requires using MFA-extracted durations in the length regulator; ignoring this causes duration mismatches that artificially inflate error.
• The dataset is small (2,335 samples) and single-speaker, which increases overfitting risk and limits generalizability compared to standard multi-speaker TTS benchmarks.

Evidence (verbatim from paper)

To evaluate the energy and pitch in the generated speech, the same methods as in the FastSpeech 2 (Ren et al., 2021) papers are used. To evaluate the pitch, the standard deviation, skewness and kurtosis for the pitch moments in the generated speech is calculated (Niebuhr and Skarnitzl, 2019; Andreeva et al., 2014). Ideally this would be as similar as possible to our ground truth. Additionally the average Dynamic Time Warping (DTW) distance of the pitch is also computed with the help of the dtwpython package (Giorgino, 2009). To evaluate the energy, the MAE between the generated speech and the ground truth is being calculated. Similar to Ren et al. (2021), the durations extracted by the MFA are used in the length regulator to ensure the same duration in the generated audio as the ground truth to calculate the MAE of the energy in the generated speech.

Citation

@misc{jeuris2022libris2s,
  title={LibriS2S: A German-English Speech-to-Speech Translation Corpus},
  author={Jeuris et al. (2022)},
  year={2022},
  note={arXiv:2204.10593}
}

• arXiv: 2204.10593