mt-quality-estimation-eval
What do Large Language Models Need for Machine Translation Evaluation? — Qian et al. (2024) (arXiv:2410.03278, 2024)
What this evaluates
Evaluates the ability of LLMs to predict human-assigned Direct Assessment (DA) scores for machine translation outputs. It probes how well different prompting strategies (zero-shot, CoT, few-shot) and input components (source, reference, error words) correlate with human judgments across various language pairs.
Datasets
Metrics
Spearman $ ho$ (primary) — range: [-1, 1]
- Spearman rank correlation coefficient between the model's predicted quality scores and the true human-assigned DA scores (averaged across annotators). Measures monotonic relationship between predicted and human scores.
Input / output format
Input: Source sentence, machine-translated output, and optionally a reference translation, error words, and annotation guidelines, formatted according to specific prompt templates (T1-T8).
Output: A single numerical quality score (DA score). Models must output a valid number; outputs without a score are discarded as 'dropped rows'.
Scoring recipe
def compute_spearman_rho(predictions, gold_scores):
# Filter out any predictions that are not valid numbers
valid_pairs = [(p, g) for p, g in zip(predictions, gold_scores) if is_valid_number(p)]
pred_vals = [p for p, g in valid_pairs]
gold_vals = [g for p, g in valid_pairs]
# Compute Spearman rank correlation
rho = scipy.stats.spearmanr(pred_vals, gold_vals).correlation
return rho
Common pitfalls
- LLMs frequently fail to generate a valid numerical score, requiring rows to be dropped before correlation calculation, which can bias results if the failure rate is high or non-random.
- High-resource language pairs (e.g., EN-DE, EN-ZH) exhibit skewed score distributions toward higher values, which can artificially lower Spearman correlation compared to medium/low-resource pairs.
- CoT prompting can degrade performance on smaller models (<10B parameters) while helping larger ones, making model size a critical confounding factor.
Evidence (verbatim from paper)
Table 3: Spearman $
ho$ correlation scores achieved by zero-shot inference using Templates 1-6 (T1-6) on various open-source LLMs for each language pair (LP). D -> rows dropped as LLM generated output without a score.
Citation
@misc{qian2024whatdo,
title={What do Large Language Models Need for Machine Translation Evaluation?},
author={Qian et al. (2024)},
year={2024},
note={arXiv:2410.03278}
}
1---2name: mt-quality-estimation-eval3description: Evaluates the ability of LLMs to predict human-assigned Direct Assessment (DA) scores for machine translation outputs. It probes how well different prompting strategies (zero-shot, CoT, few-shot) and input components (source, reference, error words) correlate with human judgments across various language pairs. Use when the user wants to benchmark on WMT QE / DA dataset (EN-DE, EN-MR, EN-ZH, ET-EN, NE-EN, RO-EN, RU-EN, SI-EN), or asks about evaluating this task. Reports Spearman $ ho$.4---56# mt-quality-estimation-eval78> What do Large Language Models Need for Machine Translation Evaluation? — Qian et al. (2024) (arXiv:2410.03278, 2024)910## What this evaluates1112Evaluates the ability of LLMs to predict human-assigned Direct Assessment (DA) scores for machine translation outputs. It probes how well different prompting strategies (zero-shot, CoT, few-shot) and input components (source, reference, error words) correlate with human judgments across various language pairs.1314## Datasets1516- **WMT QE / DA dataset (EN-DE, EN-MR, EN-ZH, ET-EN, NE-EN, RO-EN, RU-EN, SI-EN)** — total ?; splits: test (-1); repo https://github.com/surrey-nlp/LLM4MT_eval1718## Metrics1920- `Spearman $
ho$` **(primary)** — range: [-1, 1]21 - Spearman rank correlation coefficient between the model's predicted quality scores and the true human-assigned DA scores (averaged across annotators). Measures monotonic relationship between predicted and human scores.2223## Input / output format2425**Input**: Source sentence, machine-translated output, and optionally a reference translation, error words, and annotation guidelines, formatted according to specific prompt templates (T1-T8).2627**Output**: A single numerical quality score (DA score). Models must output a valid number; outputs without a score are discarded as 'dropped rows'.2829## Scoring recipe3031```python32def compute_spearman_rho(predictions, gold_scores):33 # Filter out any predictions that are not valid numbers34 valid_pairs = [(p, g) for p, g in zip(predictions, gold_scores) if is_valid_number(p)]35 pred_vals = [p for p, g in valid_pairs]36 gold_vals = [g for p, g in valid_pairs]37 # Compute Spearman rank correlation38 rho = scipy.stats.spearmanr(pred_vals, gold_vals).correlation39 return rho40```4142## Common pitfalls4344- LLMs frequently fail to generate a valid numerical score, requiring rows to be dropped before correlation calculation, which can bias results if the failure rate is high or non-random.45- High-resource language pairs (e.g., EN-DE, EN-ZH) exhibit skewed score distributions toward higher values, which can artificially lower Spearman correlation compared to medium/low-resource pairs.46- CoT prompting can degrade performance on smaller models (<10B parameters) while helping larger ones, making model size a critical confounding factor.4748## Evidence (verbatim from paper)4950> Table 3: Spearman $
ho$ correlation scores achieved by zero-shot inference using Templates 1-6 (T1-6) on various open-source LLMs for each language pair (LP). D -> rows dropped as LLM generated output without a score.5152## Citation5354```bibtex55@misc{qian2024whatdo,56 title={What do Large Language Models Need for Machine Translation Evaluation?},57 author={Qian et al. (2024)},58 year={2024},59 note={arXiv:2410.03278}60}61```6263- arXiv: 2410.03278
qhjqhj00