Bstc Eval

bstc-eval

BSTC: A Large-Scale Chinese-English Speech Translation Dataset — Zhang et al. (2021) (arXiv:2104.03575, 2021)

What this evaluates

Evaluates Chinese-to-English speech translation accuracy and real-time simultaneous interpretation latency. It probes a model's ability to handle noisy ASR inputs, segment speech into meaningful units, and produce fluent translations under strict delay constraints.

Datasets

• BSTC — total ?; splits: train (37901), dev (-1), test (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram overlap between predicted and reference translations, computed using the multi-bleu.pl script. Predicted segments are concatenated into a single long sentence before scoring.
• AL — range: other
  • Average Lagging: measures the average delay between the source and target tokens in simultaneous translation.
• CW — range: other
  • Consecutive Wait: measures the maximum number of consecutive source words waited before producing a target word.

Input / output format

Input: Mandarin audio or transcripts, processed through an ASR module and a sentence segmentation module to yield segmented Chinese text.

Output: English translation text.

Scoring recipe

# Speech Translation (concatenate segmented outputs)
pred_full = " ".join([mt_model.translate(seg) for seg in segments])
bleu = multi_bleu_score(pred_full, references)

# Simultaneous Translation
al = compute_average_lagging(source_tokens, target_tokens)
cw = compute_consecutive_wait(source_tokens, target_tokens)

Common pitfalls

• The dev set uses only 1 reference per utterance while the test set uses 4, causing a significant BLEU gap that is not due to model performance.
• Simultaneous translation metrics (AL and CW) are highly sensitive to the segmentation threshold hyperparameter (δ), requiring careful tuning to balance latency and quality.

Evidence (verbatim from paper)

We use the “multi-bleu.pl” 111111[https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/generic/multi-bleu.perl "" ] script to evaluate the BLEU score. ... We use BLEU to evaluate the translation quality and use average lagging (AL) Ma et al. (2019) and Consecutive Wait (CW) Gu et al. (2017) as latency metrics.

Citation

@misc{zhang2021bstc,
  title={BSTC: A Large-Scale Chinese-English Speech Translation Dataset},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2104.03575}
}

• arXiv: 2104.03575