Simultaneous S2st Eval

simultaneous-s2st-eval

Simultaneous Speech-to-Speech Translation Without Aligned Data — Labiausse et al. (2026) (arXiv:2602.11072, 2026)

What this evaluates

Evaluates simultaneous speech-to-speech translation models on translation accuracy, latency, speaker voice preservation, and audio naturalness across multiple languages. It probes the model's ability to generate high-quality target speech in real-time without relying on word-level aligned training data.

Datasets

• Audio-NTREX-4L — total 1200; splits: valid (600), test (600)
• Europarl-ST — total 8192; splits: valid (4096), test (4096)

Metrics

• ASR-BLEU (primary) — range: [0, 100]
  • BLEU score computed on Whisper-medium transcribed generated speech versus the reference translation, after normalizing both hypothesis and reference texts to reduce ASR error impact.
• LAAL — range: seconds
  • Length-Adaptive Average Lagging, approximating the average time (seconds) between a source word and its translation. Computed as (1/n_max) * sum(d_i - (i-1)*gamma) for i=1 to n_max, where gamma = source_duration / max(n_gen, n_ref) and n_max is the index where emission time d_i >= source_duration.
• Speaker Similarity — range: [-1, 1]
  • Cosine similarity between the source and generated speech embeddings using the WavLM Large model. Values in tables are scaled by 100.
• End Offset — range: seconds
  • Time difference in seconds between the end of the last generated word and the end of the last word from the source utterance.
• ASR-COMET — range: [0, 1]
  • COMET score (using XCOMET-XL) computed on Whisper-medium transcribed generated speech versus the reference translation.

Input / output format

Input: Multilingual source speech audio (French, Spanish, Portuguese, German, or Italian).

Output: Target English speech audio and a corresponding text stream.

Scoring recipe

def compute_metrics(generated_audio, source_audio, reference_text):
    # ASR-BLEU
    hyp_text = normalize(whisper.transcribe(generated_audio, model='medium'))
    ref_text = normalize(reference_text)
    asr_bleu = compute_bleu(hyp_text, ref_text)
    # LAAL
    timestamps = whisper.get_word_timestamps(generated_audio)
    source_dur = get_duration(source_audio)
    n_gen = len(timestamps)
    n_ref = len(reference_text.split())
    gamma = source_dur / max(n_gen, n_ref)
    n_max = next(i for i, t in enumerate(timestamps) if t >= source_dur)
    laal = sum(timestamps[i] - (i-1)*gamma for i in range(1, n_max+1)) / n_max
    return asr_bleu, laal

Common pitfalls

• ASR-BLEU scores are heavily influenced by Whisper's transcription accuracy; failing to normalize hypothesis and reference texts as specified will inflate error rates.
• LAAL requires precise word-level emission timestamps from the ASR model; using a different ASR model or varying sampling rates will break the formula's assumptions.
• Speaker similarity values in the tables are scaled by 100 (cosine similarity × 100), which may confuse readers expecting the standard [-1, 1] range.

Evidence (verbatim from paper)

We evaluate translation quality by transcribing generated speech using Whisper medium (Radford et al., 2023) and computing BLEU (Post, 2018) and COMET (Rei et al., 2020) scores with respect to a reference translation, referred to as ASR-BLEU and ASR-COMET. To reduce the impact of ASR errors, hypothesis and reference texts are normalized before computing BLEU scores.

Citation

@misc{labiausse2026simultaneousspeech,
  title={Simultaneous Speech-to-Speech Translation Without Aligned Data},
  author={Labiausse et al. (2026)},
  year={2026},
  note={arXiv:2602.11072}
}

• arXiv: 2602.11072