Seamlessm4t Human Eval

seamlessm4t-human-eval

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation — Seamless Communication et al. (2023) (arXiv:2308.11596, 2023)

What this evaluates

Probes the semantic preservation and audio naturalness of speech-to-text and speech-to-speech translation systems across 24+ languages. Uses human annotators to score translations on a 1-5 scale for meaning similarity (XSTS) and speech quality/naturalness (MOS).

Datasets

• FLEURS test partition — total ?; splits: test (-1); HF facebook/fleurs

Metrics

• XSTS (primary) — range: [1, 5]
  • Semantic similarity score on a 1-5 Likert scale. Annotators evaluate source-target pairs; median of 3 scores is taken, with additional annotators if disagreement ≥2. Scores are calibrated using a cross-lingual calibration set.
• MOS — range: [1, 5]
  • Mean Opinion Score on a 1-5 Likert scale assessing clarity, sound quality, and naturalness. Median of annotator scores is collected per item. No calibration or tie-breaking annotators are used.

Input / output format

Input: Source audio (or text) and target audio (or text) pairs from the FLEURS test set, filtered to sentences with recordings in both languages. Includes cross-lingual calibration items.

Output: A single integer score from 1 to 5 per item, representing semantic similarity (XSTS) or audio quality/naturalness (MOS).

Scoring recipe

def aggregate_scores(raw_annotator_scores):
    # raw_annotator_scores: list of lists of 1-5 scores per item
    medians = [median(scores) for scores in raw_annotator_scores]
    # Calibrate using cross-lingual calibration set bias
    calibrated = [m - calibration_bias for m in medians]
    return mean(calibrated)

Common pitfalls

• Annotators must be explicitly instructed to ignore non-speech tags (e.g., ) and pauses/noises during XSTS scoring, otherwise scores drop artificially.
• MOS evaluations are not calibrated and only cover a limited subset of language directions into English, so they cannot be directly compared to full-direction XSTS results.
• Using mean instead of median for annotator scores reduces robustness to outlier annotators, contrary to the paper's protocol.

Evidence (verbatim from paper)

For the S2ST task, we evaluate using two protocols: XSTS for translation quality, and MOS to assess naturalness. XSTS [Licht et al., 2022] evaluates translation quality in terms of semantic meaning preservation, and has previously been used to evaluate the NLLB models [NLLB Team et al., 2022]. While XSTS was originally designed to evaluate text, the protocol is effectively modality agnostic, and we required only small adaptations in order to support S2ST and S2TT tasks.

Citation

@misc{seamlesscommunication2023seamlessm4t,
  title={SeamlessM4T: Massively Multilingual & Multimodal Machine Translation},
  author={Seamless Communication et al. (2023)},
  year={2023},
  note={arXiv:2308.11596}
}

• arXiv: 2308.11596