Russian Speech Prosody Eval

russian-speech-prosody-eval

A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models — Borodin et al. (2025) (arXiv:2507.13563, 2025)

What this evaluates

Evaluates the quality, phonetic accuracy, and prosodic fidelity of Russian speech datasets and generative models across synthesis, restoration, and denoising tasks. It probes natural intonation, stress accuracy, and audio clarity using standardized subjective ratings and automatic speech quality metrics.

Datasets

• Balalaika (Proposed) — total ?; splits: test (-1)
• M-AILABS Russian — total ?; splits: test (-1)
• RUSLAN — total ?; splits: test (-1)
• Russian LibriSpeech — total ?; splits: test (-1)
• SOVA RuYoutube — total ?; splits: test (-1)
• Mozilla Common Voice 21.0 — total ?; splits: test (-1)

Metrics

• MOS (primary) — range: other
  • Mean Opinion Score on a 0-5 scale assessing studio quality, artifacts, noise, and intelligibility. Final score is the mean across annotators with a 95% confidence interval.
• IntMOS — range: other
  • Intonation MOS on a 0-5 scale evaluating natural conversation-like speech, stress accuracy, pauses, and robotic vs human likeness.
• NISQA (NMOS) — range: other
  • Automatic metric using the original NISQA architecture to predict Noiseness, Coloration, Discontinuity, Loudness, and Mean Opinion Score.
• CER — range: percent
  • Character Error Rate computed using Damerau-Levenshtein distance between reference text and ASR output (GigaAMv2-RNNT).
• PESQ — range: other
  • Perceptual Evaluation of Speech Quality, standard objective metric for speech distortion and quality.
• STOI — range: other
  • Short-Time Objective Intelligibility measure, predicts speech intelligibility.

Input / output format

Input: Audio recordings (synthetic, restored, or denoised) and corresponding ground-truth text transcripts.

Output: Numerical scores (0-5 for subjective MOS/IntMOS, continuous for NISQA/CER/PESQ/STOI) and aggregated statistics (mean, 95% CI).

Scoring recipe

def compute_mos(annotator_ratings):
    # annotator_ratings: list of lists, each inner list has ratings from >=7 annotators
    medians = [median(r) for r in annotator_ratings]
    mean_score = mean(medians)
    std_dev = std(medians)
    n = len(medians)
    ci_95 = 1.96 * (std_dev / sqrt(n))
    return mean_score, ci_95

def compute_cer(ref_text, hyp_text):
    return damerau_levenshtein_distance(ref_text, hyp_text) / len(ref_text)

Common pitfalls

• Using the wrong NISQA variant (NISQA-S was used for filtering, but original NISQA must be used for evaluation to avoid bias).
• Averaging raw annotator scores instead of taking the median per audio first, then averaging across annotators.
• Failing to ensure strict train/test separation in denoising experiments, leading to data leakage.
• Confusing TMR (Text Match Rate) with CER; TMR is percentage match, CER is edit distance.

Evidence (verbatim from paper)

To evaluate the quality of the datasets, we employed a combination of automatic and human feedback metrics. To calculate the automatic metrics, we utilized the NISQA model, which calculates the following metrics: Noiseness (NOI), Coloration (COL), Discontinuity (DIS), Loudness (LOU), and Mean Opinion Score (NMOS).

Citation

@misc{borodin2025balalaika,
  title={A Data-Centric Framework for Addressing Phonetic and Prosodic Challenges in Russian Speech Generative Models},
  author={Borodin et al. (2025)},
  year={2025},
  note={arXiv:2507.13563}
}

• arXiv: 2507.13563