Cultural Aware Mt Eval

cultural-aware-mt-eval

Benchmarking Machine Translation with Cultural Awareness — Yao et al. (2023) (arXiv:2305.14328, 2023)

What this evaluates

Evaluates machine translation systems on culturally specific items (CSIs) to measure how well they preserve cultural nuances, entities, and meanings compared to reference translations. It probes both automated lexical/semantic alignment and human judgment on translation accuracy for culturally grounded content.

Datasets

• Wikipedia Cultural Parallel Corpus — total ?; splits: test (-1)

Metrics

• CSI-Match (primary) — range: [0, 1]
  • Identifies translated culturally specific items (CSIs) in system outputs using awesome-align word alignment, then compares them against reference CSI translations from Wikidata using FuzzyWuzzy to calculate Levenshtein distance. Scores are averaged across aligned items.
• BLEU — range: [0, 100]
  • Standard n-gram precision metric with brevity penalty for sentence-level translation quality.
• BLEURT — range: [0, 100]
  • BERT-based evaluation metric trained on human judgments to capture semantic similarity between source and translation.
• COMET — range: [0, 100]
  • Cross-lingual semantic similarity metric using multilingual encoders to score translation quality against reference.
• Human Accuracy — range: percent
  • Categorizes translation accuracy into four levels: Correct (precise match), Copy (source string copied), Understandable (conveys key meaning), Wrong (entirely incorrect). Weighted average or percentage reported.

Input / output format

Input: English source sentence containing culturally specific items (CSIs) and corresponding Chinese reference translation.

Output: Chinese translation of the source sentence.

Scoring recipe

# CSI-Match Calculation
aligned_pred = awesome_align(source, system_output)
aligned_ref = awesome_align(source, reference)
scores = []
for p, r in zip(aligned_pred, aligned_ref):
    dist = levenshtein_distance(p, r)
    scores.append(1 - dist / max(len(p), len(r)))
return mean(scores)

# Human Evaluation Scoring
category_scores = {"Correct": 1.0, "Copy": 0.5, "Understandable": 0.25, "Wrong": 0.0}
human_score = sum(category_scores[annotator_label] for sample in test_set) / len(test_set)

Common pitfalls

• Standard metrics like BLEU and COMET often fail to capture cultural nuances, leading to overestimation of translation quality for CSIs.
• Word alignment tools may misalign culturally specific entities, causing CSI-Match to undercount or misalign reference items.
• Human evaluation relies on a single bilingual annotator without reported inter-annotator agreement, which may introduce subjectivity.

Evidence (verbatim from paper)

Therefore, we propose a fine-grained evaluation metric called CSI-Match, which first identifies translated CSIs in the system outputs by a word-alignment tool awesome-align and uses a fuzzy string match tool FuzzyWuzzy to compare against the reference CSI translations from Wikidata by caculating Levenshtein distance. We randomly select 10% of samples from our collected dataset and engage a bilingual annotator, who is also a native Chinese speaker, to assess the accuracy of CSI translations. To facilitate a thorough analysis, we categorize the level of accuracy into the following four distinct groups. We also provide a few examples to educate the annotator before evaluation

Citation

@misc{yao2023benchmarking,
  title={Benchmarking Machine Translation with Cultural Awareness},
  author={Yao et al. (2023)},
  year={2023},
  note={arXiv:2305.14328}
}

• arXiv: 2305.14328