Wmt Slt 22 Eval

wmt-slt-22-eval

Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22 — Tarrés et al. (2022) (arXiv:2212.01140, 2022)

What this evaluates

Evaluates the capability of sign language to text translation systems on low-resource datasets. It measures how accurately a model can convert 3D pose sequences of sign language into corresponding spoken language text.

Datasets

• FocusNews — total ?; splits: dev (-1), test (-1)
• SRF — total ?; splits: dev (-1), test (-1)

Metrics

• BLEU (primary) — range: [0, 1]
  • Standard machine translation metric computing geometric mean of n-gram precisions (1-4) with sentence-level smoothing, normalized to [0, 1].

Input / output format

Input: 3D MediaPipe keypoint sequences representing sign language videos, preprocessed to 25fps.

Output: Natural language text sentences corresponding to the signed utterance.

Scoring recipe

def compute_bleu(predictions, references):
    scores = []
    for pred, refs in zip(predictions, references):
        score = sentence_bleu([refs], pred, smoothing_function=smoothing)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Dev set performance does not correlate with official test set rankings (e.g., best dev model failed on test).
• BLEU optimization may not align with human perception; chrF++ correlates better with human rankings but yields lower scores.
• Low-resource nature leads to fundamentally poor BLEU scores despite architectural improvements.

Evidence (verbatim from paper)

We optimized our systems to obtain the best BLEU metric, without taking other metrics into consideration. However, organizers also compute chrF++ (Popovic, 2017) and BLEURT (Sellam et al., 2020) metrics (Müller et al., 2022). We find that the BLEURT score shows a similar performance than BLEU. However, for the chrF++ metric, which correlates better with respect to human relative rankings, our models score lower compared to other submissions.

Citation

@misc{tarres2022tackling,
  title={Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22},
  author={Tarrés et al. (2022)},
  year={2022},
  note={arXiv:2212.01140}
}

• arXiv: 2212.01140