Soft Pairwise Accuracy

soft-pairwise-accuracy

Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy — Thompson et al. (2024) (arXiv:2409.09598, 2024)

What this evaluates

Evaluates the reliability and discriminative power of automatic machine translation metrics by comparing their statistical significance against human MQM judgments. It measures how well a metric's pairwise system rankings align with human preferences using permutation-based p-values rather than hard binary decisions.

Datasets

• WMT Metrics Shared Task 2022/2023 — total ?; splits: test (-1); repo https://github.com/google-research/mt-metrics-eval

Metrics

• Soft Pairwise Accuracy (SPA) (primary) — range: [0, 1]
  • Aggregates continuous confidence levels (p-values) from paired permutation tests between system-level metric scores and human MQM judgments, rather than binarizing pairwise comparisons.
• Pairwise Accuracy (PA) — range: [0, 1]
  • Binarizes pairwise comparisons between system-level metric scores and human judgments, discarding uncertainty information.
• Pearson r — range: [-1, 1]
  • Measures ranking stability by computing the correlation between a metric's ranking on a subset of systems and its ranking on all systems.

Input / output format

Input: System-level scores for multiple MT systems on a test set, alongside human MQM quality judgments for the same systems.

Output: p-values for pairwise comparisons between systems, aggregated into SPA or PA scores, and ranking stability measured as change in Pearson r.

Scoring recipe

# 1. Compute system-level scores by averaging segment-level scores for each MT system.
# 2. For each pair of systems (A, B):
#    a. Run paired permutation test (1000 random permutations) on scores vs human judgments.
#    b. Extract p-value.
#    c. For PA: binarize p-value (e.g., p < 0.05 → 1, else 0).
#    d. For SPA: retain continuous p-value as confidence weight.
# 3. Aggregate pairwise results to compute final SPA/PA score.
# 4. Compute ranking stability: 1 - |Pearson_r(subset_ranking) - Pearson_r(full_ranking)|.

Common pitfalls

• Averaging segment-level scores to compute system-level metrics assumes linear behavior, which is false for metrics like BLEU and chrF.
• Caching permutations across system pairs breaks independence between p-value computations, though the authors argue it is inconsequential for SPA's confidence estimation.
• Using only 1000 permutations introduces slight randomness in SPA values across different runs.

Evidence (verbatim from paper)

Values are averaged over 1000 random trials. We find SPA to be more stable than PA in all cases.

Citation

@misc{thompson2024softpairwiseaccuracy,
  title={Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy},
  author={Thompson et al. (2024)},
  year={2024},
  note={arXiv:2409.09598}
}

• arXiv: 2409.09598