Graphpb Mos Eval

graphpb-mos-eval

GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis — Sun et al. (2020) (arXiv:2012.02626, 2020)

What this evaluates

Evaluates the naturalness and prosody quality of synthesized Chinese speech by measuring how closely the generated audio matches human-like pausing and rhythm. It probes the model's ability to capture hierarchical syntactic-semantic dependencies for prosody boundary prediction in text-to-speech systems.

Datasets

• Databaker dataset — total 10000; splits: test (100)

Metrics

• MOS (primary) — range: [0, 5]
  • Mean Opinion Score scaled from 0 to 5 in 0.5 increments. Calculated as the average rating given by at least 10 native speakers per sentence, averaged across 100 randomly selected test sentences.

Input / output format

Input: Text (graphemes/phonemes) with prosodic boundary labels (PPH/IPH edges) and corresponding acoustic features (80-dim mel-spectrograms).

Output: Synthesized audio waveform (generated via Griffin-Lim vocoder from mel-spectrograms) for subjective listening evaluation.

Scoring recipe

# For each of the 100 test sentences:
# 1. Generate audio using the model.
# 2. Have ≥10 native speakers listen and rate on a 0-5 scale (0.5 steps).
# 3. Compute average rating per sentence.
# 4. Average the per-sentence ratings across all 100 sentences to get final MOS.
# Report MOS ± variance.

Common pitfalls

• MOS is highly subjective and depends on the specific crowd-sourcing platform and rater pool; results may not generalize across different listener demographics.
• The evaluation only uses 100 test sentences, which may not capture full dataset variability or long-form prosody issues.
• Variance is reported but not statistically tested (e.g., confidence intervals or p-values), making robustness claims qualitative.

Evidence (verbatim from paper)

The subjective evaluation metric chosen in this paper is Mean Opinion Score (MOS), scaling from 0 - 5 with stages increased by 0.5. The listening tests are rated by 50 native speakers on 100 randomly chosen test sentences. Each sentence is scored by at least 10 raters.

Citation

@misc{sun2020graphpb,
  title={GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis},
  author={Sun et al. (2020)},
  year={2020},
  note={arXiv:2012.02626}
}

• arXiv: 2012.02626