Cross Lingual Pronoun Prediction Eval

cross-lingual-pronoun-prediction-eval

Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction — Guillou et al. (2019) (arXiv:1911.12091, 2019)

What this evaluates

Evaluates systems' ability to predict target-language pronoun class labels from source-language pronouns using lemmatized, POS-tagged translations and word alignments. It probes cross-lingual anaphora resolution and functional ambiguity handling in machine translation pipelines.

Datasets

• WMT 2016 Cross-lingual Pronoun Prediction Task — total ?; splits: test (-1)

Metrics

• macro-averaged recall (primary) — range: [0, 1]
  • Recall averaged across all classes, computed as the mean of per-class recall values. Ranges from 0 to 1, where 1 is perfect and 1/C is the trivial baseline for C classes.
• accuracy — range: [0, 1]
  • Standard classification accuracy, calculated as the proportion of correctly predicted instances out of the total. Noted as sensitive to class imbalance.

Input / output format

Input: Source-language pronouns, lemmatized and POS-tagged target translations, and word alignments.

Output: Predicted target-language pronoun class label for each instance.

Scoring recipe

def macro_recall(predictions, gold, num_classes):
    recalls = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return sum(recalls) / num_classes

Common pitfalls

• Using accuracy or standard F1 as the primary metric can be misleading due to class imbalance in the test set.
• Interpreting absolute accuracy values without comparing them to a dataset-specific majority-class baseline.
• Assuming a random classifier baseline is 0.5; it is actually 1/C where C is the number of classes.

Evidence (verbatim from paper)

While in 2015 we used macro-averaged $F_{1}$ as an official evaluation measure, this year we adopted macro-averaged recall, which was also recently adopted by some other competitions, e.g., by SemEval-2016 Task 4 (Nakov et al., 2016). Moreover, as in 2015, we also report accuracy as a secondary evaluation measure. Macro-averaged recall ranges in [0, 1], where a value of 1 is achieved by the perfect classifier, and a value of 0 is achieved by the classifier that mis-classifies all examples.

Citation

@misc{guillou2019wmt2016pronoun,
  title={Findings of the 2016 WMT Shared Task on Cross-lingual Pronoun Prediction},
  author={Guillou et al. (2019)},
  year={2019},
  note={arXiv:1911.12091}
}

• arXiv: 1911.12091