Word Level Qe Eval

word-level-qe-eval

An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers — Ranasinghe et al. (2021) (arXiv:2106.00143, 2021)

What this evaluates

This benchmark evaluates a model's ability to perform word-level quality estimation across multiple language pairs, identifying whether translated words are correct ('OK') or incorrect ('BAD'), as well as detecting target gaps and source-side error triggers. It probes cross-lingual transfer and fine-grained alignment-aware error detection in machine translation.

Datasets

• WMT QE datasets (En-Zh, En-Cs, En-De, En-Ru, En-Lv, De-En) — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Calculated as the product of the F1-scores for the 'OK' and 'BAD' classes against true labels independently. This applies to target words, target gaps, and source words, though the paper primarily reports the target word F1-score.

Input / output format

Input: Source sentence, target sentence, and word-level alignments for quality estimation.

Output: Per-word classification labels: 'OK' or 'BAD' for target words, target gaps, and source words.

Scoring recipe

def compute_metric(predictions, gold):
    # predictions and gold are aligned lists of 'OK'/'BAD' labels
    f1_ok = f1_score(gold, predictions, pos_label='OK')
    f1_bad = f1_score(gold, predictions, pos_label='BAD')
    return f1_ok * f1_bad

Common pitfalls

• The metric multiplies the F1-scores of the 'OK' and 'BAD' classes rather than using a standard macro or micro average, which can significantly penalize models that are unbalanced between the two classes.
• WMT evaluation criteria changed around 2019: prior to WMT 2019, gaps and target words were scored separately, while after WMT 2019 they are combined into a single result. Papers must clarify which convention they follow.

Evidence (verbatim from paper)

For evaluation, we used the approach proposed in the WMT shared tasks in which the classification performance is calculated using the multiplication of F1-scores for the 'OK' and 'BAD' classes against the true labels independently: words in the target ('OK' for correct words, 'BAD' for incorrect words), gaps in the target ('OK' for genuine gaps, 'BAD' for gaps indicating missing words) and source words ('BAD' for words that lead to errors in the target, 'OK' for other words) (Specia et al., 2018). In recent WMT shared tasks, the most popular category was predicting quality for words in the target. Therefore, in Section 5 we only report the F1-score for words in the target.

Citation

@misc{ranasinghe2021multilingual,
  title={An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers},
  author={Ranasinghe et al. (2021)},
  year={2021},
  note={arXiv:2106.00143}
}

• arXiv: 2106.00143