Matcha Tts Eval

matcha-tts-eval

Matcha-TTS: A fast TTS architecture with conditional flow matching — Mehta et al. (2023) (arXiv:2309.03199, 2023)

What this evaluates

Evaluates the synthesis speed, intelligibility, and naturalness of a non-autoregressive TTS model trained with conditional flow matching on English speech. It measures how efficiently the model converts text to audio and how closely the output matches human perception of naturalness.

Datasets

• LJ Speech — total ?; splits: test (-1); repo https://keithito.com/LJ-Speech-Dataset/

Metrics

• RTF — range: other
  • Real-Time Factor: the ratio of wall-clock synthesis time to the duration of the generated audio, including vocoding time. Lower values indicate faster synthesis.
• WER — range: percent
  • Word Error Rate: computed by transcribing the synthesized audio with the Whisper medium ASR system and comparing the output to the ground truth text. Lower values indicate higher intelligibility.
• MOS (primary) — range: [1, 5]
  • Mean Opinion Score: average rating from a crowdsourced listening test where 80 native English speakers rate naturalness on a 1–5 integer scale. Higher values indicate better perceived naturalness.

Input / output format

Input: Input graphemes converted to IPA phones via phonemizer (espeak-ng backend), representing the text to be synthesized.

Output: Synthesized audio waveform (generated by the acoustic model followed by HiFi-GAN vocoder).

Scoring recipe

def compute_metrics(predictions, gold):
    # RTF
    rtf = synthesis_wall_clock_time / generated_audio_duration
    # WER
    asr_text = whisper_medium_model.transcribe(predictions)
    wer = compute_wer(asr_text, gold.text)
    # MOS
    ratings = [listener_rating(predictions) for _ in range(80)]
    mos = mean(ratings)
    return rtf, wer, mos

Common pitfalls

• MOS values are highly context-dependent (listener demographics, instructions, loudness normalization) and should not be compared across different papers.
• RTF and quality are coupled via ODE solver steps (NFE); reporting speed without specifying NFE is misleading.
• WER is computed using an external ASR (Whisper medium) rather than direct phoneme alignment, which may not perfectly reflect human intelligibility.

Evidence (verbatim from paper)

We evaluated our approach both objectively and subjectively. ... After training the systems, we assessed the synthesis speed and intelligibility of the different conditions, by computing the real time factor (RTF) mean and standard deviation when synthesising the test set, and evaluating the word error rate (WER) when applying the Whisper medium ASR system to the results, since the WERs of strong ASR systems correlate well with intelligibility ... To evaluate the naturalness of the synthesised audio we ran a mean opinion score (MOS) listening test. We selected 40 utterances (4 groups of 10) of different lengths from the test set and synthesised each utterance using all conditions, loudness-normalising every stimulus using EBU R128. 80 subjects (self-reported as native English speakers using headphones) were crowdsourced through Prolific to listen to and rate these stimuli. For each stimulus, listeners were asked “How natural does the synthesised speech sound?”, and provided responses on an integer rating scale from 1 (“Completely unnatural”) to 5 (“Completely natural”) adopted from the Blizzard Challenge

Citation

@misc{mehta2023matchatts,
  title={Matcha-TTS: A fast TTS architecture with conditional flow matching},
  author={Mehta et al. (2023)},
  year={2023},
  note={arXiv:2309.03199}
}

• arXiv: 2309.03199