Ment Eval

ment-eval

Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation — Yanzhi Tian et al. (2026) (arXiv:2601.07338, 2026)

What this evaluates

Evaluates the reliability of machine translation evaluation metrics across reference-based, quality estimation, and LLM-as-a-judge paradigms when applied to non-literal content such as internet slang, idioms, and literary expressions.

Datasets

• MENT — total ?; splits: test (-1); repo https://github.com/BITHLP/RATE

Metrics

• Accuracy (Acc) — range: [0, 1]
  • Proportion of correctly ranked or scored translation pairs compared to human judgments.
• Accuracy-t (Acc-t) — range: [0, 1]
  • Thresholded accuracy metric assessing metric alignment with human scores within a specified tolerance margin.
• Pearson Correlation (r) — range: [-1, 1]
  • Linear correlation coefficient measuring the strength of the linear relationship between metric scores and human reference scores.
• Spearman correlation (ρ) — range: [-1, 1]
  • Rank-based correlation coefficient measuring the monotonic relationship between metric and human scores.
• Composite Meta Score (primary) — range: [0, 1]
  • Average of Accuracy, Accuracy-t, Pearson Correlation, and Spearman Correlation computed across both system and segment levels.

Input / output format

Input: Source sentence, reference translation, and machine-translated hypothesis (or hypothesis and reference for reference-free metrics).

Output: A scalar quality score representing the estimated translation quality.

Scoring recipe

def compute_composite_meta_score(predictions, gold):
    acc = compute_accuracy(predictions, gold)
    acc_t = compute_accuracy_t(predictions, gold)
    pearson_r = compute_pearson_correlation(predictions, gold)
    spearman_rho = compute_spearman_correlation(predictions, gold)
    return (acc + acc_t + pearson_r + spearman_rho) / 4.0

Common pitfalls

• LLM-as-a-Judge metrics suffer from knowledge cutoffs when evaluating emerging slang or idioms absent from pre-training corpora.
• Reference-free (QE) models often fail on non-literal content because they rely on literal/formal pre-training data and cannot verify semantic nuances.
• Domain-specific challenges require different mitigation strategies; e.g., search agents are needed for historical allusions in poetry, while knowledge retrieval helps with cross-cultural slang.

Evidence (verbatim from paper)

Following WMT23 Metrics Shared Task Freitag et al. ([2023]), we adopt the composite meta score to assess metric performance across both system level and segment level. Our evaluation includes Accuracy (Acc, Acc-t), Pearson Correlation ($r$), and Spearman correlation ($ ho$). The final meta score is calculated as the average of individual statistics (Feng et al., [2025b]).

Citation

@misc{tian2026beyond,
  title={Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation},
  author={Yanzhi Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.07338}
}

• arXiv: 2601.07338