qhjqhj00 wmt17-paraphrase-eval
Generative Pretraining for Paraphrase Evaluation — Weston et al. (2021) (arXiv:2107.08251, 2021)
What this evaluates
Evaluates how well a model's predicted quality scores correlate with human judgments on machine translation output. It probes semantic equivalence and paraphrase detection capabilities in the context of MT evaluation.
Datasets
- WMT17 — total ?; splits: test (-1)
Metrics
Pearson |r|(primary) — range: [0, 1]- Pearson correlation coefficient between model predictions and human judgment scores, reported as absolute value.
Kendall |τ|— range: [0, 1]- Kendall rank correlation coefficient between model predictions and human judgment scores, reported as absolute value.
Input / output format
Input: A pair of texts: a system-generated translation (or paraphrase) and a human reference translation.
Output: A single scalar score representing the predicted quality/paraphrase similarity.
Scoring recipe
def compute_metrics(predictions, human_scores):
# predictions and human_scores are aligned lists of floats
pearson_r = pearsonr(predictions, human_scores)[0]
kendall_tau = kendalltau(predictions, human_scores)[0]
return abs(pearson_r), abs(kendall_tau)
Common pitfalls
- Report absolute values of correlation coefficients, as negative correlations are flipped to positive.
- Average results across source languages to match the paper's headline numbers; per-language scores differ.
- Distinguish the fine-tuning data (WMT15/16 English subsets) from the evaluation benchmark (WMT17).
Evidence (verbatim from paper)
The metrics reported are absolute Kendall $| au|$ and Pearson $|r|$ averaged across each source language.
Citation
@misc{weston2021generative,
title={Generative Pretraining for Paraphrase Evaluation},
author={Weston et al. (2021)},
year={2021},
note={arXiv:2107.08251}
}
- arXiv: 2107.08251