Iwslt2023 St Eval

iwslt2023-st-eval

Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation — Sperber et al. (2024) (arXiv:2406.03881, 2024)

What this evaluates

Evaluates automatic speech translation systems on long-form audio across offline, multilingual, and simultaneous conditions. Probes the model's ability to handle segmentation, resegmentation, and translation quality under varying acoustic and linguistic challenges.

Datasets

• IWSLT2023 TED Test Set — total 75; splits: test (75)
• IWSLT2023 ACL Test Set — total ?; splits: test (-1)

Metrics

• COMET (primary) — range: [0, 1]
  • Reference-based neural metric using cross-lingual encoders to score translation quality. Outputs a score typically in [0, 1] or [-1, 1] depending on the model version.
• chrF — range: [0, 100]
  • Character n-gram F-score measuring overlap between hypothesis and reference at the character level.
• WER — range: [0, 1]
  • Word Error Rate used for segment alignment during resegmentation, calculated as (insertions + deletions + substitutions) / total words.

Input / output format

Input: Long-form audio recordings in English (source language), optionally with reference segment boundaries for simultaneous tasks.

Output: Target language text translations (German, Japanese, Mandarin Chinese), segmented to match reference boundaries after automatic resegmentation.

Scoring recipe

# 1. Resegment system output to align with reference segments
aligned_hypotheses = mwerSegmenter(system_output, reference_text)
# 2. Compute automatic metrics on aligned segments
comet_scores = [comet_score(hyp, ref) for hyp, ref in zip(aligned_hypotheses, references)]
chrf_scores = [chrf_score(hyp, ref) for hyp, ref in zip(aligned_hypotheses, references)]
# 3. Aggregate (e.g., mean)
final_metric = mean(comet_scores) # or chrf_scores

Common pitfalls

• TED reference translations are subtitles with compressed/omitted content, which may unfairly penalize literal translations compared to unconstrained references.
• Automatic resegmentation using WER minimization can introduce segmentation noise that degrades automatic metric correlation with human judgments.
• Simultaneous tasks require strict latency constraints (≤2s AL), which are not captured by standard offline translation metrics.

Evidence (verbatim from paper)

Results show strong correlation between direct human assessments and automatic metrics like COMET, despite segmentation noise, with COMET outperforming traditional metrics such as chrF; findings confirm that human evaluation remains essential for validating automatic metrics in speech translation, especially in noisy, long-form, and multilingual settings.

Citation

@misc{sperber2024iwslt2023st,
  title={Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation},
  author={Sperber et al. (2024)},
  year={2024},
  note={arXiv:2406.03881}
}

• arXiv: 2406.03881