Wmt2023 Qe Eval

wmt2023-qe-eval

Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task — Geng et al. (2023) (arXiv:2309.13230, 2023)

What this evaluates

Evaluates machine translation quality estimation across sentence-level scoring, word-level error detection, and fine-grained error span classification. It probes a model's ability to predict translation quality and localize specific errors without relying on human references during inference.

Datasets

• WMT2022 QE EN-DE dataset — total ?; splits: train (-1), val (-1)
• WMT2022 Metric EN-DE dataset — total ?; splits: train (-1), val (-1)
• WMT17/19/20 Post-editing EN-DE datasets — total ?; splits: train (-1)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient measures the quality of binary classifications. Formula: (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for error detection. Formula: 2 * (precision * recall) / (precision + recall).
• Spearman score — range: [-1, 1]
  • Spearman's rank correlation coefficient measures the monotonic relationship between predicted quality scores and human judgments.

Input / output format

Input: Source sentence and machine-translated target sentence (hypothesis).

Output: Sentence-level quality score, word-level error tags (e.g., GOOD/BAD), and fine-grained error spans with severity classification.

Scoring recipe

def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
    mcc = (tp*tn - fp*fn) / ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    f1 = 2 * tp / (2*tp + fp + fn) if (2*tp + fp + fn) > 0 else 0
    spearman = scipy.stats.spearmanr(predictions, gold).correlation
    return {'MCC': mcc, 'F1': f1, 'Spearman': spearman}

Common pitfalls

• MCC and F1 are reported for both word-level and span-level tasks, but the paper notes they are computed after a post-processing pipeline converts word-level 'BAD' tags into spans.
• Spearman correlation is used for sentence-level ranking, which is sensitive to score calibration and may not reflect absolute quality differences.
• The evaluation relies on the WMT2023 QE Shared Task test set, which is not publicly released for independent benchmarking.

Evidence (verbatim from paper)

We achieve the best results on EN-DE for both word-level and fine-grained error span detection sub-tasks with an MCC of 29.7 (+4.1 than the second best system) and F1 score of 28.4 (+1.1) respectively. We rank 2nd place on sentence-level sub-tasks with a Spearman score of 47.9 (-0.4 than the best system).

Citation

@misc{geng2023unify,
  title={Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task},
  author={Geng et al. (2023)},
  year={2023},
  note={arXiv:2309.13230}
}

• arXiv: 2309.13230