Mt Proxy Correlation

mt-proxy-correlation

Translation as a Scalable Proxy for Multilingual Evaluation — Issaka et al. (2026) (arXiv:2601.11778, 2026)

What this evaluates

Evaluates whether machine translation quality serves as a scalable proxy for multilingual model performance on downstream tasks. It measures the alignment between MT metric scores and actual benchmark success across languages and model sizes.

Datasets

• FLORES-200 — total ?; splits: test (-1); HF flores200
• Multilingual Benchmarks (Belebele, HellaSwag, AfriMMLU, Global MMLU, TruthfulQA, INCLUDE, MGSM, MLQA, AfriXNLI) — total ?; splits: test (-1)

Metrics

• Pearson r (primary) — range: [-1, 1]
  • Measures linear correlation between MT scores and benchmark scores across languages. Computed per model-metric-benchmark triplet.
• Spearman ρ — range: [-1, 1]
  • Measures rank correlation between MT scores and benchmark scores across languages. Computed per model-metric-benchmark triplet.

Input / output format

Input: Per language, a model's MT metric score and its corresponding score on a downstream multilingual benchmark.

Output: Correlation coefficient (Pearson r or Spearman ρ) computed across languages.

Scoring recipe

def compute_correlation(mt_scores, bench_scores):
    # mt_scores: list of MT metric scores per language
    # bench_scores: list of benchmark scores per language
    r = pearsonr(mt_scores, bench_scores).statistic
    rho = spearmanr(mt_scores, bench_scores).statistic
    return {"pearson_r": r, "spearman_rho": rho}

Common pitfalls

• Correlation is computed across languages, not models or instances, so it reflects cross-lingual alignment rather than per-instance accuracy.
• Metric sensitivity varies significantly by translation dataset (FLORES-200 vs. WMT24++ vs. NTREX) and task category (semantic vs. specialized reasoning).
• Median correlations can mask high variance on specific benchmark-metric pairs, requiring careful inspection of individual coefficients.

Evidence (verbatim from paper)

We analyze the relationship between MT quality and downstream multilingual task performance by computing correlations between 7 MT metrics and 9 multilingual benchmarks across 14 LLMs. ... For most metrics, the median correlation exceeds $r=0.80$. Neural metrics are the most consistently correlated: xCOMET achieves a median $r=0.91$, MetricX a median $r=0.89$, and SSA-COMET a median $r=0.87$.

Citation

@misc{issaka2026translation,
  title={Translation as a Scalable Proxy for Multilingual Evaluation},
  author={Issaka et al. (2026)},
  year={2026},
  note={arXiv:2601.11778}
}

• arXiv: 2601.11778