# Xc Translate Eval

> Evaluates a model's ability to perform cross-cultural machine translation, specifically probing its capacity to accurately transcreate culturally nuanced entity names across multiple language pairs rather than merely transliterating or omitting them. Use when the user wants to benchmark on XC-Translate, WMT (17-21), or asks about evaluating this task. Reports M-ETA.

- Skill: `qhjqhj00/xc-translate-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/xc-translate-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/xc-translate-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/xc-translate-eval

---


# xc-translate-eval

> Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs — Conia et al. (2024) (arXiv:2410.14057, 2024)

## What this evaluates

Evaluates a model's ability to perform cross-cultural machine translation, specifically probing its capacity to accurately transcreate culturally nuanced entity names across multiple language pairs rather than merely transliterating or omitting them.

## Datasets

- **XC-Translate** — total ?; splits: test (-1)
- **WMT (17-21)** — total ?; splits: test (-1)

## Metrics

- `BLEU` — range: [0, 100]
  - Standard n-gram overlap metric between reference and hypothesis translations, typically computed with sentence-level averaging and smoothing.
- `COMET` — range: [0, 1]
  - A neural reference-based metric that uses a pre-trained model to predict human judgment scores for translation quality.
- `M-ETA` **(primary)** — range: percent
  - A custom metric measuring entity translation accuracy. Calculated as the percentage of culturally specific entity names in the source text that are correctly transcreated in the target translation.

## Input / output format

**Input**: Source sentence in English (or other source language) containing culturally specific entity names.

**Output**: Translated target sentence.

## Scoring recipe

```python
def compute_metrics(predictions, references, entities_gold):
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    comet_score = comet_model.predict(predictions, references).mean()
    correct_entities = 0
    total_entities = len(entities_gold)
    for pred, gold_ent in zip(predictions, entities_gold):
        if gold_ent in pred: # or fuzzy match
            correct_entities += 1
    m_eta = (correct_entities / total_entities) * 100 if total_entities > 0 else 0
    return {'BLEU': bleu, 'COMET': comet_score, 'M-ETA': m_eta}
```

## Common pitfalls

- BLEU and COMET scores can remain high even when entity names are completely mistranslated or left in the source language, making them unreliable for this specific cross-cultural task.
- Models often default to transliteration or zero-shot translation of entity names instead of performing cultural transcreation, which drastically lowers M-ETA scores.
- Evaluating on general-purpose MT benchmarks (like WMT) without entity annotations will mask failures in culturally nuanced translation.

## Evidence (verbatim from paper)

> Table[1] shows the results of the systems averaged over all the language pairs of XC-Translate in terms of BLEU, COMET, and M-ETA. We can first observe that MT systems, such as mBART-50, M2M-100, and NLLB-200, as well as LLMs, such as GPT-3, GPT-3.5, and GPT-4, obtain unsatisfactory M-ETAs on XC-Translate, with NLLB-200 and GPT-4 achieving the highest average score of 17.9% and 25.3%, respectively. These results support two of our hypotheses: (i) translating texts that contains challenging entity names is particularly difficult, and simply trying to translate the original entity name is often not sufficient to produce a correct translation; and (ii) BLEU and COMET are not a reliable metrics to evaluate the translation quality in this setting.

## Citation

```bibtex
@misc{conia2024xc_translate,
  title={Towards Cross-Cultural Machine Translation with Retrieval-Augmented Generation from Multilingual Knowledge Graphs},
  author={Conia et al. (2024)},
  year={2024},
  note={arXiv:2410.14057}
}
```

- arXiv: 2410.14057