Pearson Correlation

pearson-correlation

On conducting better validation studies of automatic metrics in natural language generation evaluation — Wei (2019) (arXiv:1907.13362, 2019)

What this evaluates

Evaluates the validity of automatic machine translation metrics by measuring their linear correlation with human Direct Assessment (DA) scores at both segment and system levels. It emphasizes rigorous validation protocols, including adaptive sample size determination for human judgments and statistical significance testing to compare metric performance.

Datasets

• WMT Metrics Shared Task — total ?; splits: test (-1)

Metrics

• pearson-correlation (primary) — range: [-1, 1]
  • Measures linear association between two ordinal variables (e.g., metric scores and DA scores). Formula: r_xy = sum((x_i - x_bar)(y_i - y_bar)) / (sqrt(sum((x_i - x_bar)^2)) * sqrt(sum((y_i - y_bar)^2))). Ranges from [-1, 1].
• direct-assessment — range: [1, 100]
  • Human judgment score on a continuous sliding bar from 1 to 100. Scores are averaged over many workers to approximate the population mean translation quality.

Input / output format

Input: Paired scores for each translation segment or system: the automatic metric's score and the corresponding human Direct Assessment (DA) score.

Output: Pearson correlation coefficient (r) and, when comparing metrics, a p-value from Williams' test indicating statistical significance of the difference in correlations.

Scoring recipe

def pearson_correlation(x, y):
    n = len(x)
    x_bar = sum(x) / n
    y_bar = sum(y) / n
    numerator = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y))
    denom_x = sum((xi - x_bar) ** 2 for xi in x) ** 0.5
    denom_y = sum((yi - y_bar) ** 2 for yi in y) ** 0.5
    return numerator / (denom_x * denom_y)

Common pitfalls

• High system-level correlation does not guarantee high segment-level correlation; a metric may only penalize bad outputs without distinguishing average from good ones.
• Human judgment variance varies across translations, making fixed sample sizes unreliable for Direct Assessment consistency.
• Standard p-value tests for r=0 are insufficient for comparing metrics; Williams' test is required to determine if one metric significantly outperforms another.

Evidence (verbatim from paper)

The emerging consensus in WMT is the use of Pearson correlation in segment and system-level evaluation of metrics. Given n paired data points {(x_1, y_1), ..., (x_n, y_n)}, the sample Pearson correlation is defined as: r_xy = sum((x_i - x_bar)(y_i - y_bar)) / (sqrt(sum((x_i - x_bar)^2)) sqrt(sum((y_i - y_bar)^2))) where x_bar and y_bar are the samples means for x_i and y_i, respectively. This correlation measures a linear association between two ordinal variables and ranges from [-1, 1].

Citation

@misc{wei2019conducting,
  title={On conducting better validation studies of automatic metrics in natural language generation evaluation},
  author={Wei (2019)},
  year={2019},
  note={arXiv:1907.13362}
}

• arXiv: 1907.13362