Mt Reasoning Scaling Eval

mt-reasoning-scaling-eval

Test-Time Scaling of Reasoning Models for Machine Translation — Li et al. (2025) (arXiv:2510.06471, 2025)

What this evaluates

This evaluation probes the test-time scaling properties of reasoning models across diverse machine translation tasks. It measures how varying reasoning budgets and iterative self-correction workflows impact translation quality across literary, biomedical, cultural, and commonsense domains.

Datasets

• WMT24-Literary — total 43; splits: test (43)
• MetaphorTrans — total 2000; splits: test (2000)
• LitEval-Corpus — total 187; splits: test (187)
• WMT24-Biomedical — total 600; splits: test (600)
• WMT23-Biomedical — total 585; splits: test (585)
• CAMT — total 6948; splits: test (6948)
• Commonsense-MT — total 1200; splits: test (1200)
• RTT — total 100; splits: test (100)
• RAGTrans — total 1999; splits: test (1999)

Metrics

• COMET-22 (primary) — range: [0, 1]
  • Reference-based neural metric that predicts translation quality by comparing the hypothesis to the reference and source text. Scores are normalized to [0, 1].
• COMETKiwi-22 — range: [0, 1]
  • Reference-free neural metric that predicts translation quality using only the source and hypothesis. Scores are normalized to [0, 1].
• GRB — range: [0, 100]
  • LLM-as-judge metric (Gemini-2.0-Flash) that scores translation quality on a 0-100 scale using the reference text.
• GRF — range: [0, 100]
  • LLM-as-judge metric (Gemini-2.0-Flash) that scores translation quality on a 0-100 scale without using the reference text.
• GEA100 — range: [0, 100]
  • Specialized LLM-as-judge metric for literary translation that assesses style and expressiveness on a 0-100 scale.
• GEA5 — range: [1, 5]
  • Coarse-grained version of GEA that assesses literary style and expressiveness on a 1-5 scale.

Input / output format

Input: Source text for direct translation; source text plus draft translation and optionally a numerical quality score for post-editing.

Output: Target language translation (or post-edited translation).

Scoring recipe

def compute_metrics(predictions, references):
    comet22_scores = [comet22_model.predict(src=r, hyp=p) for p, r in zip(predictions, references)]
    grb_scores = [gemini_judge.score(src=r, hyp=p) for p, r in zip(predictions, references)]
    return {
        'COMET-22': mean(comet22_scores),
        'GRB': mean(grb_scores)
    }

Common pitfalls

• Applying maximum reasoning budgets to direct translation tasks degrades performance due to forced extrapolation.
• Standard reference-based metrics (COMET) fail to capture literary style nuances; specialized LLM-as-judge prompts (GEA) are required for literary benchmarks.
• Post-editing without an explicit quality score signal (QS prompt) yields minimal self-correction improvements.

Evidence (verbatim from paper)

We assess translation quality using a suite of automatic metrics, encompassing both reference-based and reference-free approaches, alongside a specialized LLM-based judge for literary texts. For a standardized assessment, we employ two variants from the COMET framework: the reference-based COMET-22 and the reference-free COMETKiwi-22. For LLM-based evaluation, we employ Gemini-2.0-Flash. We first define two general-purpose metrics, Gemini Reference-Based (GRB) and Gemini Reference-Free (GRF), which provide a quality score on a 0-100 scale.

Citation

@misc{li2025testtimescaling,
  title={Test-Time Scaling of Reasoning Models for Machine Translation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2510.06471}
}

• arXiv: 2510.06471