Pearson Correlation Coefficient

pearson-correlation-coefficient

Machine Translation Evaluation with BERT Regressor — Shimanaka et al. (2019) (arXiv:1907.12679, 2019)

What this evaluates

Probes the ability of an automated metric to correlate with human judgments of translation quality. Specifically, it tests how well a model's predicted segment-level scores align with Direct Assessment (DA) human scores across multiple language pairs.

Datasets

• WMT-2017 Metrics Shared Task — total ?; splits: train (4824), val (536), test (3920)

Metrics

• Pearson correlation coefficient (primary) — range: [-1, 1]
  • Computes the Pearson correlation coefficient between the metric's predicted scores and the Direct Assessment (DA) human evaluation scores for each translation segment.

Input / output format

Input: Source sentence and machine-translated target sentence pair.

Output: A single continuous float score representing the predicted translation quality.

Scoring recipe

def compute_pearson(predictions, gold):
    n = len(predictions)
    mean_p = sum(predictions) / n
    mean_g = sum(gold) / n
    cov = sum((p - mean_p) * (g - mean_g) for p, g in zip(predictions, gold))
    std_p = (sum((p - mean_p)**2 for p in predictions) / n) ** 0.5
    std_g = (sum((g - mean_g)**2 for g in gold) / n) ** 0.5
    return cov / (std_p * std_g)

Common pitfalls

• Evaluates only to-English language pairs, so results do not generalize to other directions.
• Uses Direct Assessment (DA) human scores as the ground truth, not BLEU or COMET.
• Segment-level evaluation requires averaging correlations across language pairs rather than pooling all segments globally.

Evidence (verbatim from paper)

We evaluated each metric using the Pearson correlation coefficient between the metric scores and the DA human scores.

Citation

@misc{shimanaka2019machine,
  title={Machine Translation Evaluation with BERT Regressor},
  author={Shimanaka et al. (2019)},
  year={2019},
  note={arXiv:1907.12679}
}

• arXiv: 1907.12679