Crossvoice S2st Eval

crossvoice-s2st-eval

CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning — Hira et al. (2024) (arXiv:2406.00021, 2024)

What this evaluates

Evaluates cross-lingual speech-to-speech translation (S2ST) systems on translation accuracy and prosody preservation. It measures how well a cascade-based S2ST pipeline preserves speaker identity and naturalness while translating speech across different language pairs.

Datasets

• CVSS-T — total ?; splits: test (-1)
• Indic-TTS — total ?; splits: test (-1)
• Fisher — total ?; splits: test (-1)
• MuST-C — total ?; splits: test (-1)
• VoxPopuli — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Computed by first transcribing the generated speech using Whisper (temperature=1, greedy decoding), then calculating the BLEU score between the generated transcript and the ground-truth reference transcript.
• MOS-c — range: [0, 5]
  • Mean Opinion Score measuring subjective perceived quality/naturalness of the synthesized speech output.

Input / output format

Input: Source speech audio in a target language (e.g., Spanish, German, Italian, Hindi, French).

Output: Translated speech audio in the source language (e.g., English).

Scoring recipe

# Transcribe generated speech for BLEU
gen_text = whisper_transcribe(generated_audio, temperature=1, decoding="greedy")
bleu = compute_bleu(reference_text, gen_text)

# Compute MOS-c
mos_c = mean([human_rating(audio) for audio in generated_audio])

Common pitfalls

• BLEU is computed on Whisper-transcribed audio rather than direct text, so ASR errors directly impact the translation metric.
• SOTA BLEU scores are taken from original papers (BLEU-r) instead of being re-evaluated under identical conditions, risking unfair comparison.
• MOS scores are subjective human ratings; the paper reports mean ± std, so statistical variance must be considered when comparing systems.

Evidence (verbatim from paper)

For calculating the BLEU scores, we employed Whisper (using the temperature setting of one and greedy decoding) for generating transcripts of the speech generated using CrossVoice and SOTA methods. MOS-c score is almost the same as MOS-h (i.e., the GT) and also beats MOS-v scores of the vanilla TTS considerably, by almost 40% on each task.

Citation

@misc{hira2024crossvoice,
  title={CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning},
  author={Hira et al. (2024)},
  year={2024},
  note={arXiv:2406.00021}
}

• arXiv: 2406.00021