Mtqe Generation Based Eval

mtqe-generation-based-eval

LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods — Cui et al. (2025) (arXiv:2505.16129, 2025)

What this evaluates

Evaluates machine translation quality estimation (MTQE) methods by measuring how well their segment-level scores correlate with human judgments across multiple language pairs. It specifically tests a generation-based paradigm where LLMs create reference translations instead of directly scoring outputs.

Datasets

• WMT22 Test Sets (8 language pairs) — total ?; splits: test (-1)

Metrics

• Spearman rank correlation (ρ) (primary) — range: [-1, 1]
  • Standard Spearman rank correlation coefficient measuring the monotonic relationship between predicted quality scores and human judgment scores.
• Pearson correlation (r) — range: [-1, 1]
  • Standard Pearson product-moment correlation coefficient measuring the linear relationship between predicted quality scores and human judgment scores.

Input / output format

Input: Source sentence, machine-translated target sentence, and evaluation prompt.

Output: A continuous quality score (float) representing estimated translation quality, or a generated reference translation used for subsequent embedding-based similarity scoring.

Scoring recipe

def compute_spearman(pred_scores, human_scores):
    n = len(pred_scores)
    pred_ranks = [sorted(pred_scores).index(x) + 1 for x in pred_scores]
    human_ranks = [sorted(human_scores).index(x) + 1 for x in human_scores]
    d_sq = sum((p - h)**2 for p, h in zip(pred_ranks, human_ranks))
    return 1 - (6 * d_sq) / (n * (n**2 - 1))

Common pitfalls

• Comparing reference-free methods against reference-based metrics, which introduces different semantic inputs and violates fair comparison paradigms.
• Assuming larger LLM parameters guarantee better scoring stability or accuracy; the paper shows smaller models (e.g., LLaMA-3-8B) can outperform larger/unstable ones.
• Using direct LLM scoring (regression) instead of the proposed generation-based reference creation, which yields significantly lower correlation.

Evidence (verbatim from paper)

Empirical results across 8 LLMs and 8 language pairs show significantly higher segment-level correlation with human judgments (up to +68% improvement) compared to direct scoring baselines, demonstrating that LLMs excel at fluent generation and semantic alignment when used for reference creation rather than direct regression. Table 2 reports results for ρ and r across UK-EN, CS-EN, RU-EN, and DE-EN.

Citation

@misc{cui2025llmsarenotscorers,
  title={LLMs Are Not Scorers: Rethinking MT Evaluation with Generation-Based Methods},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2505.16129}
}

• arXiv: 2505.16129