Streamuni Eval

streamuni-eval

StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model — Shoutao Guo et al. (2025) (arXiv:2507.07803, 2025)

What this evaluates

Evaluates real-time speech translation systems on latency and translation quality across multiple language pairs. It probes the model's ability to dynamically decide when to generate and truncate translations while processing streaming audio inputs.

Datasets

• MuST-C English→German — total ?; splits: test (-1)
• MuST-C English→Spanish — total ?; splits: test (-1)
• CoVoST2 English→Chinese — total ?; splits: test (-1)
• CoVoST2 French→English — total ?; splits: test (-1)

Metrics

• SacreBLEU (primary) — range: other
  • Corpus-level BLEU score computed using the official SacreBLEU toolkit, which handles tokenization and reference normalization automatically.
• COMET (primary) — range: [0, 1]
  • A neural machine translation quality estimation metric that scores the semantic similarity between the system translation and the reference translation.
• Average Lagging (AL) — range: other
  • Measures the average delay between the end of the input audio and the generation of each output word, calculated in milliseconds.
• Length-Adaptive Average Lagging (LAAL) — range: other
  • A normalized version of AL that accounts for the length of the reference translation, providing a more comparable latency metric across different output lengths.
• Stream SacreBLEU — range: other
  • A streaming variant of SacreBLEU that evaluates translation quality using alignments generated by mWERSegmenter to handle document-level streaming outputs.
• StreamLAAL — range: other
  • The streaming-specific latency metric used for StreamST, derived from LAAL but computed using mWERSegmenter alignments for document-level inputs.

Input / output format

Input: Streaming audio chunks (320ms for En-Zh, 640ms for other directions) accompanied by a prompt specifying the target language.

Output: Text containing the transcribed audio followed by the translation, separated by a token.

Scoring recipe

def evaluate(predictions, references, audio_timestamps, task):
    if task == 'SimulST':
        al = simul_eval.compute_average_lagging(predictions, references, audio_timestamps)
        sacrebleu = sacrebleu.corpus_bleu(predictions, [references])
        comet = comet_model.predict(predictions, references)
        return {'AL': al, 'SacreBLEU': sacrebleu, 'COMET': comet}
    elif task == 'StreamST':
        alignments = mwer_segmenter.align(predictions, references)
        stream_sacrebleu = sacrebleu.corpus_bleu(predictions, [references], alignment=alignments)
        stream_laal = compute_stream_laal(predictions, references, audio_timestamps, alignments)
        return {'Stream SacreBLEU': stream_sacrebleu, 'StreamLAAL': stream_laal}

Common pitfalls

• Confusing SimulST and StreamST evaluation setups; SimulST uses SimulEval directly, while StreamST requires mWERSegmenter alignment preprocessing before computing metrics.
• Latency metrics (AL/LAAL) are highly sensitive to chunk size and wait-k policy settings, which vary by language direction (320ms vs 640ms) and must be reported consistently.
• Stream SacreBLEU and StreamLAAL are task-specific variants that differ from standard SimulST metrics and require specific alignment handling for document-level inputs.

Evidence (verbatim from paper)

In evaluating streaming generation systems, we need to assess two critical aspects: latency and generation quality. To quantify latency, we utilize the Average Lagging (AL) (Ma et al. [2019]) and Length-Adaptive Average Lagging (LAAL) metrics (Papi et al. [2022]), which measures the delay between input reception and output generation. For translation quality, we use the SacreBLEU (Post [2018]) and COMET (Rei et al. [2022]) metrics. For the SimulST task, we employ the SimulEval tool (Ma et al. [2020]) to evaluate our StreamUni. In the StreamST task, we follow the setup of Papi et al. ([2024]). We first use mWERSegmenter (Matusov et al. [2005]) for aligning document-level translation with references and then convert these alignments into consistent metrics used in the SimulST task. In this task, the latency metric is termed StreamLAAL, and translation quality is assessed using Stream SacreBLEU.

Citation

@misc{guo2025streamuni,
  title={StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model},
  author={Shoutao Guo et al. (2025)},
  year={2025},
  note={arXiv:2507.07803}
}

• arXiv: 2507.07803