Xc Translate Eval

xc-translate-eval

Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs — Conia et al. (2024) (arXiv:2410.14057, 2024)

What this evaluates

Evaluates a model's ability to perform cross-cultural machine translation, specifically probing its capacity to accurately transcreate culturally nuanced entity names across multiple language pairs rather than merely transliterating or omitting them.

Datasets

• XC-Translate — total ?; splits: test (-1)
• WMT (17-21) — total ?; splits: test (-1)

Metrics

• BLEU — range: [0, 100]
  • Standard n-gram overlap metric between reference and hypothesis translations, typically computed with sentence-level averaging and smoothing.
• COMET — range: [0, 1]
  • A neural reference-based metric that uses a pre-trained model to predict human judgment scores for translation quality.
• M-ETA (primary) — range: percent
  • A custom metric measuring entity translation accuracy. Calculated as the percentage of culturally specific entity names in the source text that are correctly transcreated in the target translation.

Input / output format

Input: Source sentence in English (or other source language) containing culturally specific entity names.

Output: Translated target sentence.

Scoring recipe

def compute_metrics(predictions, references, entities_gold):
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    comet_score = comet_model.predict(predictions, references).mean()
    correct_entities = 0
    total_entities = len(entities_gold)
    for pred, gold_ent in zip(predictions, entities_gold):
        if gold_ent in pred: # or fuzzy match
            correct_entities += 1
    m_eta = (correct_entities / total_entities) * 100 if total_entities > 0 else 0
    return {'BLEU': bleu, 'COMET': comet_score, 'M-ETA': m_eta}

Common pitfalls

• BLEU and COMET scores can remain high even when entity names are completely mistranslated or left in the source language, making them unreliable for this specific cross-cultural task.
• Models often default to transliteration or zero-shot translation of entity names instead of performing cultural transcreation, which drastically lowers M-ETA scores.
• Evaluating on general-purpose MT benchmarks (like WMT) without entity annotations will mask failures in culturally nuanced translation.

Evidence (verbatim from paper)

Table[1] shows the results of the systems averaged over all the language pairs of XC-Translate in terms of BLEU, COMET, and M-ETA. We can first observe that MT systems, such as mBART-50, M2M-100, and NLLB-200, as well as LLMs, such as GPT-3, GPT-3.5, and GPT-4, obtain unsatisfactory M-ETAs on XC-Translate, with NLLB-200 and GPT-4 achieving the highest average score of 17.9% and 25.3%, respectively. These results support two of our hypotheses: (i) translating texts that contains challenging entity names is particularly difficult, and simply trying to translate the original entity name is often not sufficient to produce a correct translation; and (ii) BLEU and COMET are not a reliable metrics to evaluate the translation quality in this setting.

Citation

@misc{conia2024xc_translate,
  title={Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs},
  author={Conia et al. (2024)},
  year={2024},
  note={arXiv:2410.14057}
}

• arXiv: 2410.14057