Canmt Eval

canmt-eval

Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation — Yuan et al. (2026) (arXiv:2604.24361, 2026)

What this evaluates

Evaluates large language models and specialized MT systems on culture-aware machine translation across 12 language pairs. It probes the models' ability to preserve cultural nuances and adapt to explicit semantic versus communicative translation constraints.

Datasets

• CanMT — total ?; splits: test (-1); repo https://github.com/zkyuan-scir/CanMT-a-Culture-Aware-Novel-Driven-Parallel-Dataset-for-Machine-Translation

Metrics

• translation performance (primary) — range: [1, 5]
  • A 1-to-5 scale rating assessing translation quality, fluency, and cultural appropriateness. Scores are averaged across 12 language pairs to compute an overall average.

Input / output format

Input: Source sentence or passage from literary novels in English, Spanish, Japanese, or Russian.

Output: Translated sentence or passage in the target language (English, Spanish, Japanese, Chinese, or Russian).

Scoring recipe

def compute_metric(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        # Rated on a 1-5 scale for fluency, adequacy, and cultural appropriateness
        score = llm_judge_score(pred, ref)
        scores.append(score)
    return sum(scores) / len(scores) if scores else 0.0

Common pitfalls

• Models often demonstrate a 'knowledge-application gap', recognizing cultural knowledge but failing to operationalize it in the final translation.
• LLM-as-a-judge evaluations can be unreliable without reference translations to ground the scoring criteria.
• Performance heavily depends on the translation paradigm constraint (semantic vs. communicative) applied in the prompt.

Evidence (verbatim from paper)

Table 4: Overall translation performance across language directions. For the Qwen3 series and Seed-X models, only the non-reasoning variants are included.

Citation

@misc{yuan2026culturemt,
  title={Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation},
  author={Yuan et al. (2026)},
  year={2026},
  note={arXiv:2604.24361}
}

• arXiv: 2604.24361