Cultural Nuance Mt Eval

cultural-nuance-mt-eval

"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs — Van Doren et al. (2026) (arXiv:2602.04729, 2026)

What this evaluates

This benchmark evaluates how well multilingual LLMs preserve cultural nuance, idioms, puns, and culturally embedded concepts during machine translation. It probes the persistent gap between grammatical accuracy and cultural resonance by measuring translation quality across different figurative and non-figurative segment categories.

Datasets

• Cultural Nuance MT Benchmark — total ?; splits: test (-1)

Metrics

• overall quality (primary) — range: [0, 3]
  • Human annotators assign an integer score from 0 to 3 to each translated segment, where 0 indicates poor quality and 3 indicates excellent quality. The metric is averaged across segments, models, and languages.

Input / output format

Input: Source text segments containing idioms, puns, holidays, or culturally embedded concepts in various languages.

Output: Translated text segments in the target language.

Scoring recipe

# Pseudo-code for scoring protocol
scores = []
for segment in dataset:
    translation = model.translate(segment.source)
    if translation is omitted:
        rating = "NA"
    else:
        rating = human_annotate(translation, scale=[0,1,2,3], dimensions=["overall", "audience", "style", "content"])
        scores.append(rating["overall"])
avg_score = mean(scores)
# Statistical comparison uses CLMM with Tukey-adjusted post-hoc tests

Common pitfalls

• Omissions (rated NA) are excluded from the CLMM quality modeling but significantly impact coverage metrics, so they must be tracked separately.
• Inter-rater reliability drops substantially for idioms and puns compared to holidays, meaning lower scores may reflect annotator uncertainty rather than pure model failure.
• Language effects interact with segment category; low scores for certain languages are driven disproportionately by figurative language, not uniform degradation.

Evidence (verbatim from paper)

All scores are reported on a 0-3 ordinal scale, where higher values indicate better translation quality. Table 2 Average rating on a 0-3 (4-point) ordinal scale by model across languages of overall translation quality, appropriateness to intended audience, faithfulness to style of the original, and faithfulness to content of the original.

Citation

@misc{vandoren2026bemycheese,
  title={"Be My Cheese?": Cultural Nuance Benchmarking for Machine Translation in Multilingual LLMs},
  author={Van Doren et al. (2026)},
  year={2026},
  note={arXiv:2602.04729}
}

• arXiv: 2602.04729