Rakugo Listening Test Eval

rakugo-listening-test-eval

How Similar or Different Is Rakugo Speech Synthesizer to Professional Performers? — Kato et al. (2020) (arXiv:2010.11549, 2020)

What this evaluates

Evaluates the perceptual quality of synthesized rakugo speech by comparing it to professional human performances across multiple dimensions, including naturalness, character distinguishability, content understandability, entertainment value, and overall skill level. The benchmark probes whether TTS systems can capture the nuanced performance modeling required for traditional Japanese verbal entertainment.

Datasets

• Misomame — total 1; splits: test (1)

Metrics

• Mean Opinion Score (MOS) (primary) — range: [1, 5]
  • Five-point Likert scale ratings (1–5) averaged across 292 listeners for five dimensions: naturalness, character distinguishability, content understandability, entertainment, and rakugo skill level. Scores are computed per dimension and then averaged across all listeners.

Input / output format

Input: Audio recordings of the rakugo story 'Misomame', either synthesized sentence-by-sentence with fixed pauses matching the real recording, or performed by professional rakugo artists at three skill ranks (zenza, futatsume, shin-uchi). All audio is normalized to -26 dBov.

Output: Five numerical ratings on a 5-point scale corresponding to: 1) naturalness, 2) character distinguishability, 3) content understandability, 4) entertainment, 5) rakugo skill level.

Scoring recipe

ratings = []
for listener in listeners:
    ratings.append([
        listener.score('naturalness'),
        listener.score('character_distinguishability'),
        listener.score('content_understandability'),
        listener.score('entertainment'),
        listener.score('skill_level')
    ])
mos = np.mean(ratings, axis=0)
return mos

Common pitfalls

• Synthesized speech pauses are fixed to match the real recording rather than being predicted, which may artificially inflate naturalness scores but does not reflect full synthesis capability.
• The evaluation relies entirely on subjective MOS ratings from a general listener pool rather than expert acoustic analysis, making results sensitive to listener fatigue and demographic composition.
• Character distinguishability and content understandability are highly correlated (r=0.538), meaning poor character modeling directly impacts perceived story comprehension, complicating isolated metric optimization.

Evidence (verbatim from paper)

We asked listeners to answer a five-scale mean opinion score (MOS) based test. Listeners listened to either speech by the professional performers (zenza, futatsume, or shin-uchi) or the synthesized speech, and they evaluated them according to the five questions below. 1) How natural did the performer sound? 2) How accurately did you think you could distinguish each character? 3) How well did you think you could understand the content? 4) How well were you entertained? 5) How high was the rakugo skill level of the performer.

Citation

@misc{kato2020rakugo,
  title={How Similar or Different Is Rakugo Speech Synthesizer to Professional Performers?},
  author={Kato et al. (2020)},
  year={2020},
  note={arXiv:2010.11549}
}

• arXiv: 2010.11549