qhjqhj00 wmt-mt-meta-eval
Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! — Perrella et al. (2024) (arXiv:2408.13831, 2024)
What this evaluates
Evaluates machine translation metrics by measuring their alignment with human judgments at segment and system levels. It probes whether metrics can correctly rank translations and systems based on quality, and tests the robustness of meta-evaluation statistics like correlation and pairwise ranking accuracy.
Datasets
- WMT Test Sets — total ?; splits: test (-1)
Metrics
System-level pairwise ranking accuracy— range: [0, 1]- Measures the percentage of system pairs where the metric's ranking matches the human judgment ranking.
Pearson correlation(primary) — range: [-1, 1]- Measures the linear correlation between metric scores and human judgment scores. Calculated as the covariance of the scores divided by the product of their standard deviations.
Segment-level pairwise ranking accuracy with tie calibration— range: [0, 1]- Evaluates the metric's ability to rank segments in the same order as human judgments, with adjustments for correctly predicting ties.
Input / output format
Input: Source segment, machine-translated candidate, and human judgment scores (scalar) for the candidate.
Output: A scalar quality score assigned to each translation (segment-level) or system (system-level) by the evaluated metric.
Scoring recipe
def compute_pairwise_accuracy(metric_scores, human_scores):
correct = 0
total = 0
for i in range(len(metric_scores)):
for j in range(i + 1, len(metric_scores)):
if human_scores[i] != human_scores[j]:
total += 1
if (metric_scores[i] > metric_scores[j]) == (human_scores[i] > human_scores[j]):
correct += 1
return correct / total if total > 0 else 0.0
def compute_pearson(metric_scores, human_scores):
n = len(metric_scores)
mean_m = sum(metric_scores) / n
mean_h = sum(human_scores) / n
cov = sum((m - mean_m) * (h - mean_h) for m, h in zip(metric_scores, human_scores))
std_m = (sum((m - mean_m)**2 for m in metric_scores))**0.5
std_h = (sum((h - mean_h)**2 for h in human_scores))**0.5
return cov / (std_m * std_h) if std_m * std_h != 0 else 0.0
Common pitfalls
- Segment-level correlations must be calculated after grouping translations by source segment; otherwise, scores are artificially inflated.
- Tie calibration should not be conducted on the test set, as it introduces evaluation bias.
- Metrics optimized to mimic human judgments may exploit spurious correlations rather than learning robust translation quality.
Evidence (verbatim from paper)
Specifically, metrics are evaluated at two granularity levels: at the segment level, metrics assign a score to every translation, and they are ranked according to their ability to discern between higher- and lower-quality translations; at the system level, metrics assign a score to each MT system, and they are ranked according to their ability to discern between superior and inferior systems. At both granularity levels, metrics can be evaluated using several statistical methods, such as the Kendall τ and Pearson ρ correlation coefficients, which have traditionally been applied at the segment and system levels, respectively. A final metrics ranking is derived by aggregating results from all the chosen statistics. For example, at WMT23, the final ranking was computed from the following three statistics: 1. System-level pairwise ranking accuracy Kocmi et al. ([2021]), which evaluates metrics based on their ability to rank systems in the same order as human judgments. 2. System- and segment-level Pearson correlation, which measures the degree to which metric scores and human scores are correlated linearly. 3. Segment-level pairwise ranking accuracy with tie calibration Deutsch et al. (
Citation
@misc{perrella2024guardians,
title={Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!},
author={Perrella et al. (2024)},
year={2024},
note={arXiv:2408.13831}
}
- arXiv: 2408.13831