Mexpresso Mdral S2st Eval

mexpresso-mdral-s2st-eval

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation — Min-Jae Hwang et al. (2024) (arXiv:2406.02733, 2024)

What this evaluates

Evaluates noise-robust expressive speech-to-speech translation (S2ST) across English-Spanish and Spanish-English directions. It measures how well systems preserve naturalness and expressive style when translating speech under clean and artificially noisy conditions.

Datasets

• mExpresso / mDRAL — total ?; splits: dev (-1), test (-1)

Metrics

• Naturalness MOS (primary) — range: other
  • Mean Opinion Score from human listeners rating the naturalness of synthesized speech on a typical 1-5 scale. Reported as mean ± 95% confidence interval.
• S-MOS — range: other
  • Style/Expressivity Mean Opinion Score from human listeners rating how well the synthesized speech preserves the source speaker's vocal style, emotion, and tone. Reported as mean ± 95% confidence interval.

Input / output format

Input: Source speech audio (clean or noisy) in English or Spanish.

Output: Target language speech waveform (24 kHz) synthesized from XLS-R 10K units via DINO-PRETSSEL and HiFi-GAN vocoder.

Scoring recipe

def compute_metrics(predictions, gold, conditions):
    # predictions: list of generated audio waveforms
    # gold: list of reference utterances
    # conditions: list of 'clean' or 'noisy' labels
    ratings = []
    for pred, ref, cond in zip(predictions, gold, conditions):
        nat_score = human_rate_naturalness(pred)
        style_score = human_rate_style(pred, ref)
        ratings.append({'naturalness': nat_score, 'style': style_score, 'condition': cond})
    
    clean_nat = [r['naturalness'] for r in ratings if r['condition'] == 'clean']
    noisy_nat = [r['naturalness'] for r in ratings if r['condition'] == 'noisy']
    clean_style = [r['style'] for r in ratings if r['condition'] == 'clean']
    noisy_style = [r['style'] for r in ratings if r['condition'] == 'noisy']
    
    return {
        'clean_naturalness_mos': mean(clean_nat),
        'noisy_naturalness_mos': mean(noisy_nat),
        'clean_s_mos': mean(clean_style),
        'noisy_s_mos': mean(noisy_style)
    }

Common pitfalls

• Confusing 'Clean' vs 'Noisy' source conditions, as results are reported separately for each condition rather than averaged.
• Ignoring the 95% confidence intervals when claiming statistical superiority between systems.
• Assuming evaluation noise matches training SNR (6dB-40dB), as test noise levels are not explicitly detailed in the protocol.

Evidence (verbatim from paper)

Naturalness MOS↑ | S-MOS↑ | ... Table 2: Subjective evaluation results for various expressive S2ST systems with a 95% confidence interval. The highest scores are in bold typeface. "Clean" and "Noisy" denote that the source speech of S2ST system was clean and noisy, respectively.

Citation

@misc{hwang2024textless,
  title={Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation},
  author={Min-Jae Hwang et al. (2024)},
  year={2024},
  note={arXiv:2406.02733}
}

• arXiv: 2406.02733