Transevalnia Eval

transevalnia-eval

TransEvalnia: Reasoning-based Evaluation and Ranking of Translations — Sproat et al. (2025) (arXiv:2507.12724, 2025)

What this evaluates

Probes a model's ability to rank machine translation candidates by quality and provide fine-grained, dimensionally structured justifications aligned with MQM standards. It also evaluates the model's robustness to candidate ordering (position bias) when performing comparative translation assessment.

Datasets

• WMT-2024 en-es — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• WMT-2023 en-de — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• WMT-2023 zh-en — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• WMT-2022 en-ru — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• WMT-2021 en-ja — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• WMT-2021 ja-en — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• Hard en-ja — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• Generic — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• Haiku 100 — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia
• Haiku Full — total ?; splits: test (-1); repo https://github.com/SakanaAI/TransEvalnia

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of translation pairs or triplets where the model correctly identifies the human-preferred candidate.
• Bias inconsistency — range: [1, p]
  • B = \sum_{i=1}^{n} \frac{|b_i|}{n}, where n is the number of source sentences and |b_i| is the cardinality of the set of 'best' translations across all candidate order permutations.

Input / output format

Input: Source sentence paired with two or three candidate machine translations. Candidates are presented in varying orders (permutations) to assess position bias.

Output: Ranked translation (best candidate), fine-grained reasoning/justification per evaluation dimension (e.g., accuracy, terminology, audience appropriateness), and an overall Likert score.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(predictions)

def compute_bias_inconsistency(preds_by_perm, n):
    total = 0
    for i in range(n):
        best_set = set()
        for perm_preds in preds_by_perm[i]:
            best_set.add(perm_preds[0])
        total += len(best_set)
    return total / n

Common pitfalls

• Models are highly sensitive to the order in which translation candidates are presented; failing to use interleaved or order-agnostic prompting yields inflated accuracy but high position bias.
• Comparing against XCOMET-XXL or MetricX-XXL is unfair as they are fine-tuned on WMT data, potentially inflating their performance relative to out-of-domain evaluators.
• No-reasoning variants often achieve higher raw ranking accuracy but lack robustness to position bias compared to reasoning-enabled variants.

Evidence (verbatim from paper)

For all sets we ran the various TransEvalnia models using Qwen as the LLM, with Sonnet also being used in some cases. We summarize the results showing the accuracies all systems for each dataset in Figure 2. ... Bias inconsistency is computed as $B = \sum_{i=1}^{n} \frac{|b_i|}{n}$ , where $n$ is the number of source sentences, and $|b_i|$ is the cardinality of the set of 'best' translations, which can range from 1 to $p$, the number of permutations.

Citation

@misc{sproat2025transevalnia,
  title={TransEvalnia: Reasoning-based Evaluation and Ranking of Translations},
  author={Sproat et al. (2025)},
  year={2025},
  note={arXiv:2507.12724}
}

• arXiv: 2507.12724