# Pearson Correlation

> Evaluates the validity of automatic machine translation metrics by measuring their linear correlation with human Direct Assessment (DA) scores at both segment and system levels. It emphasizes rigorous validation protocols, including adaptive sample size determination for human judgments and statistical significance testing to compare metric performance. Use when the user has predictions and gold and needs to compute pearson-correlation.

- Skill: `qhjqhj00/pearson-correlation` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/pearson-correlation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/pearson-correlation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/pearson-correlation

---


# pearson-correlation

> On conducting better validation studies of automatic metrics in natural language generation evaluation — Wei (2019) (arXiv:1907.13362, 2019)

## What this evaluates

Evaluates the validity of automatic machine translation metrics by measuring their linear correlation with human Direct Assessment (DA) scores at both segment and system levels. It emphasizes rigorous validation protocols, including adaptive sample size determination for human judgments and statistical significance testing to compare metric performance.

## Datasets

- **WMT Metrics Shared Task** — total ?; splits: test (-1)

## Metrics

- `pearson-correlation` **(primary)** — range: [-1, 1]
  - Measures linear association between two ordinal variables (e.g., metric scores and DA scores). Formula: r_xy = sum((x_i - x_bar)(y_i - y_bar)) / (sqrt(sum((x_i - x_bar)^2)) * sqrt(sum((y_i - y_bar)^2))). Ranges from [-1, 1].
- `direct-assessment` — range: [1, 100]
  - Human judgment score on a continuous sliding bar from 1 to 100. Scores are averaged over many workers to approximate the population mean translation quality.

## Input / output format

**Input**: Paired scores for each translation segment or system: the automatic metric's score and the corresponding human Direct Assessment (DA) score.

**Output**: Pearson correlation coefficient (r) and, when comparing metrics, a p-value from Williams' test indicating statistical significance of the difference in correlations.

## Scoring recipe

```python
def pearson_correlation(x, y):
    n = len(x)
    x_bar = sum(x) / n
    y_bar = sum(y) / n
    numerator = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y))
    denom_x = sum((xi - x_bar) ** 2 for xi in x) ** 0.5
    denom_y = sum((yi - y_bar) ** 2 for yi in y) ** 0.5
    return numerator / (denom_x * denom_y)
```

## Common pitfalls

- High system-level correlation does not guarantee high segment-level correlation; a metric may only penalize bad outputs without distinguishing average from good ones.
- Human judgment variance varies across translations, making fixed sample sizes unreliable for Direct Assessment consistency.
- Standard p-value tests for r=0 are insufficient for comparing metrics; Williams' test is required to determine if one metric significantly outperforms another.

## Evidence (verbatim from paper)

> The emerging consensus in WMT is the use of Pearson correlation in segment and system-level evaluation of metrics. Given n paired data points {(x_1, y_1), ..., (x_n, y_n)}, the sample Pearson correlation is defined as: r_xy = sum((x_i - x_bar)(y_i - y_bar)) / (sqrt(sum((x_i - x_bar)^2)) sqrt(sum((y_i - y_bar)^2))) where x_bar and y_bar are the samples means for x_i and y_i, respectively. This correlation measures a linear association between two ordinal variables and ranges from [-1, 1].

## Citation

```bibtex
@misc{wei2019conducting,
  title={On conducting better validation studies of automatic metrics in natural language generation evaluation},
  author={Wei (2019)},
  year={2019},
  note={arXiv:1907.13362}
}
```

- arXiv: 1907.13362