Wmt21 Qe Eval

wmt21-qe-eval

QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task — Wang et al. (2021) (arXiv:2112.14890, 2021)

What this evaluates

Evaluates machine translation quality estimation systems by predicting human judgments on translation adequacy and fluency (Direct Assessment) and classifying translation errors (CED). It probes the model's ability to correlate predicted scores with human ratings and accurately detect translation quality issues across multiple language pairs.

Datasets

• WMT 2021 Quality Estimation Shared Task datasets — total ?; splits: test (-1), dev (-1)

Metrics

• Pearson's correlation (primary) — range: [-1, 1]
  • Measures the linear correlation between predicted quality scores and z-standardized human Direct Assessment (DA) scores.
• Matthews correlation coefficient (MCC) — range: [-1, 1]
  • Measures the correlation between predicted and actual error classification labels, accounting for true/false positives and negatives.

Input / output format

Input: Source sentence and target machine-translated sentence pair.

Output: Predicted quality score (for DA) or error classification label (for CED).

Scoring recipe

def pearson(preds, gold):
    n = len(preds)
    mp, mg = sum(preds)/n, sum(gold)/n
    cov = sum((p-mp)*(g-mg) for p,g in zip(preds, gold))
    sp = (sum((p-mp)**2 for p in preds)/n)**0.5
    sg = (sum((g-mg)**2 for g in gold)/n)**0.5
    return cov/(sp*sg) if sp*sg > 0 else 0.0

def mcc(preds, gold):
    tp = sum(1 for p,g in zip(preds,gold) if p==1 and g==1)
    fp = sum(1 for p,g in zip(preds,gold) if p==1 and g==0)
    fn = sum(1 for p,g in zip(preds,gold) if p==0 and g==1)
    tn = sum(1 for p,g in zip(preds,gold) if p==0 and g==0)
    num = tp*tn - fp*fn
    den = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    return num/den if den > 0 else 0.0

Common pitfalls

• Forgetting to z-standardize human DA scores before computing Pearson correlation, as specified in the protocol.
• Mixing up the regression task (DA) with the classification task (CED), which require different metrics and evaluation protocols.
• Using WMT 2020 test sets for final model selection instead of the blind WMT 2021 test sets specified for the shared task submission.

Evidence (verbatim from paper)

The Pearson's correlations between our model's predictions and the human DA judges (z-standardized mean DA score) are shown in Table 1. Brief results of Matthews correlations (MCC) on development sets are shown in Table 4.

Citation

@misc{wang2021qemind,
  title={QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2112.14890}
}

• arXiv: 2112.14890