Moverscore

moverscore

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance — Zhao et al. (2019) (arXiv:1909.02622, 2019)

What this evaluates

Automated evaluation of text generation quality by computing semantic distance between system outputs and human references using contextualized embeddings and Earth Mover's Distance (EMD). It probes a model's ability to capture meaning-based similarity rather than surface-level n-gram overlaps across machine translation, summarization, dialogue, and image captioning tasks.

Datasets

• WMT 2017 — total 21000; splits: test (-1)
• TAC-2008 / TAC-2009 — total 92; splits: test (-1)
• BAGEL — total 202; splits: test (-1)
• SFHOTEL — total 398; splits: test (-1)
• MS-COCO — total 5000; splits: test (-1)

Metrics

• Pearson r (primary) — range: [-1, 1]
  • Computes the Pearson correlation coefficient between the metric's scores and human-assigned quality scores across all instances in a dataset.
• Spearman ρ — range: [-1, 1]
  • Computes the Spearman rank correlation coefficient between the metric's scores and human-assigned quality scores.

Input / output format

Input: System-generated text and one or more human reference texts.

Output: A scalar Earth Mover's Distance score between the contextualized embedding distributions of the system and reference texts.

Scoring recipe

def compute_pearson_r(predictions, gold):
    n = len(predictions)
    mean_p = sum(predictions) / n
    mean_g = sum(gold) / n
    cov = sum((p - mean_p) * (g - mean_g) for p, g in zip(predictions, gold))
    std_p = (sum((p - mean_p)**2 for p in predictions) / n) ** 0.5
    std_g = (sum((g - mean_g)**2 for g in gold) / n) ** 0.5
    return cov / (std_p * std_g)

Common pitfalls

• MoverScore outputs a distance (lower is better), but the evaluation reports correlation with human scores (higher is better). Confusing the directionality is common.
• The metric's performance heavily depends on the embedding model and fine-tuning task; using pre-trained BERT without fine-tuning yields significantly lower correlation than MNLI-finetuned BERT.
• Aggregation via p-means and n-gram granularity (unigram vs bigram) substantially impact results, and defaults are not explicitly standardized across all tasks.

Evidence (verbatim from paper)

We employ Pearson’s r and Spearman’s ρ to measure the correlation. We consider two variants of MoverScore: word mover and sentence mover, described below. In all language pairs, the best correlation is achieved by our word mover metrics that use a BERT pretrained on MNLI as the embedding generator and PMeans to aggregate the embeddings from different BERT layers.

Citation

@misc{zhao2019moverscore,
  title={MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance},
  author={Zhao et al. (2019)},
  year={2019},
  note={arXiv:1909.02622}
}

• arXiv: 1909.02622