Realsi Sst Eval

realsi-sst-eval

Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice — Cheng et al. (2025) (arXiv:2507.17527, 2025)

What this evaluates

Evaluates end-to-end simultaneous speech-to-speech translation quality and latency on long-form, multi-domain continuous speech. Probes the model's ability to maintain semantic accuracy, speaker voice characteristics, and low delay in real-time multilingual dialogue.

Datasets

• RealSI — total ?; splits: test (-1)

Metrics

• VIP (primary) — range: [0, 100] (score)
  • Human evaluation metric measuring the proportion of valid semantic fragments that accurately convey the speaker’s original intent, closely aligning with human interpreter judgments.
• SVIP — range: [0, 100] (score)
  • Speech-specific extension of VIP measuring the proportion of valid speech semantic fragments within a session, requiring core information accuracy, acceptable latency, appropriate pace, and acoustic clarity.
• BLEURT — range: [0, 100] or [-1, 1]
  • Automated neural metric for translation quality based on BERT representations.
• COMET — range: [0, 100] or [-1, 1]
  • Automated neural metric for translation quality using cross-lingual representations.
• FLAL — range: [0, ∞) seconds
  • First Letter Appearance Lagging: time until the system outputs the first determined translation at the paragraph level.
• AL — range: [0, ∞) seconds
  • Average Lagging: widely adopted sentence-level latency metric measuring average delay between source and target utterances.
• LAAL — range: [0, ∞) seconds
  • Length Adaptive Average Lagging: sentence-level latency metric normalized by output length.

Input / output format

Input: Continuous audio speech in Chinese or English (approx. 5 minutes per sample for long-form; sentence-level for short-form).

Output: Translated audio speech in the target language (with voice cloning) or translated text, generated simultaneously as the source audio plays.

Scoring recipe

def calculate_vip_svip(transcript_segments):
    valid = 0
    total = len(transcript_segments)
    for seg in transcript_segments:
        if (conveys_core_intent(seg) and
            latency_acceptable(seg) and
            pace_acceptable(seg) and
            clarity_acceptable(seg)):
            valid += 1
    return (valid / total) * 100

def calculate_latency(source_timestamps, target_timestamps):
    al = mean(target_timestamps - source_timestamps)
    flal = target_timestamps[0] - source_timestamps[0]
    return al, flal

Common pitfalls

• Automated metrics like BLEURT and COMET may not fully reflect true translation capabilities compared to human VIP/SVIP scores.
• Commercial baselines often employ post-hoc rewriting strategies, making direct latency/quality comparisons with human-like simultaneous interpreters misleading.
• Missing table entries indicate lack of feature support (e.g., speech-to-speech) rather than zero performance.

Evidence (verbatim from paper)

For text translation quality assessment, we primarily rely on the idea of the human evaluation metric, Valid Information Proportion (VIP) [[6]], which measures how accurately the translation output conveys the speaker’s original intent for each semantic fragment, closely aligning with human interpreter judgments.

Citation

@misc{cheng2025seedliveinterpret2,
  title={Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice},
  author={Cheng et al. (2025)},
  year={2025},
  note={arXiv:2507.17527}
}

• arXiv: 2507.17527