Wqe Metric Eval

wqe-metric-eval

Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement — Sarti et al. (2025) (arXiv:2505.23183, 2025)

What this evaluates

Evaluates the ability of unsupervised and supervised metrics to identify word-level translation errors by comparing their continuous scores against human-annotated error spans and multi-annotator agreement rates.

Datasets

• QE4PE — total ?; splits: test (-1)
• DivEMT — total ?; splits: test (-1)

Metrics

• Average Precision (AP) (primary) — range: [0, 1]
  • Area under the precision-recall curve computed over continuous metric scores against binary human error labels. Measures overall ranking quality across all thresholds.
• F1* — range: [0, 1]
  • The maximum F1 score obtained by iterating over all possible thresholds to binarize continuous metric scores into positive/negative labels matching human annotations.
• Spearman correlation — range: [-1, 1]
  • Rank correlation between continuous WQE metric scores and human edit counts (number of annotators marking a token as an error) across multiple annotation sets.

Input / output format

Input: Continuous scores from WQE metrics for each token in a machine-translated sentence, paired with human-annotated binary error labels or multi-annotator edit counts.

Output: Binary classification (error/non-error) or continuous score, depending on the metric evaluation phase.

Scoring recipe

def compute_metrics(scores, gold_binary, edit_counts=None):
    ap = average_precision_score(gold_binary, scores)
    f1_star = max(f1_score(gold_binary, scores > t) for t in np.linspace(0, 1, 1000))
    if edit_counts is not None:
        spearman_rho = spearmanr(scores, edit_counts).correlation
    return ap, f1_star, spearman_rho

Common pitfalls

• Random baseline AP values match the proportion of tokens marked as errors, which can vary greatly across datasets and languages.
• Default xcomet metrics show high precision but very low recall (32-26%), which is problematic for WQE applications where omitting errors causes oversights.
• Metric rankings are highly unstable when evaluated against a single annotator's labels due to subjective variability.

Evidence (verbatim from paper)

We report Average Precision (AP) as a general measure of metric quality across the full score range, and we estimate calibrated metric performance as the best F1 score (F1∗) across all thresholds for binarizing continuous metric scores into pos./neg. labels matching human annotation.

Citation

@misc{sarti2025unsupervised,
  title={Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement},
  author={Sarti et al. (2025)},
  year={2025},
  note={arXiv:2505.23183}
}

• arXiv: 2505.23183