Wmt22 Slt Eval

wmt22-slt-eval

Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation — Dey et al. (2022) (arXiv:2210.13326, 2022)

What this evaluates

Evaluates sign language translation from video to spoken text. It probes the model's ability to handle long videos, large vocabularies, and high singleton rates by leveraging full-body and lip-reading visual features.

Datasets

• WMT 2022 Shared Task — total ?; splits: dev (-1), test (-1)
• PHOENIX 2014T — total ?; splits: dev (-1), test (-1)

Metrics

• BLEU (primary) — range: [0, 1]
  • Standard n-gram precision with brevity penalty. The paper also reports 'reduced BLEU (RedB)', which filters out singleton words (appearing only once in training) to mitigate vocabulary sparsity.

Input / output format

Input: Per-frame visual feature embeddings extracted from video frames (768-dim mouth patches via AV-HuBERT or full-body via I3D).

Output: Sequence of spoken German words.

Scoring recipe

def compute_bleu(preds, refs, filter_singletons=False):
    if filter_singletons:
        preds = [w for w in preds if w not in singletons]
        refs = [[w for w in r if w not in singletons] for r in refs]
    return standard_bleu_score(preds, refs)

Common pitfalls

• Ignoring singleton words (appearing only once in training) severely degrades performance on this dataset.
• Utterance boundary markers (full stops) significantly impact BLEU scores (~1% relative difference).
• Lip-reading features improve rare word prediction but may hurt performance on common words without lexical data augmentation.

Evidence (verbatim from paper)

Configurations are evaluated on the WMT 2022 Dev dataset using the reduced BLEU (RedB) and standard BLEU (Stand.) score as metric.

Citation

@misc{dey2022clean,
  title={Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation},
  author={Dey et al. (2022)},
  year={2022},
  note={arXiv:2210.13326}
}

• arXiv: 2210.13326