Cult Eval Eval

cult-eval-eval

From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation — Han et al. (2026) (arXiv:2603.17303, 2026)

What this evaluates

This benchmark probes a machine translation model's ability to accurately translate culture-loaded expressions (idioms, proverbs, culture-specific items) while preserving their figurative, contextual, and cultural meanings. It evaluates whether models can avoid literal or superficial translations that strip away culturally grounded nuances.

Datasets

• CulT-Eval — total 7959; splits: test (-1)

Metrics

• ACRE (primary) — range: [0, 1]
  • A taxonomy-aware evaluation metric designed to systematically identify and quantify culturally induced meaning deviations. It aligns model outputs with human annotations across a five-way cultural taxonomy (Material, Social, Linguistic, Religious, Ecological) to capture nuances beyond surface-level similarity.
• Pearson r — range: [-1, 1]
  • Standard Pearson correlation coefficient measuring the linear relationship between model metric scores and human annotation scores.
• Spearman ρ — range: [-1, 1]
  • Standard Spearman rank correlation coefficient measuring the monotonic relationship between model metric scores and human annotation scores.

Input / output format

Input: Chinese source sentence containing a culture-loaded expression, paired with a verified English reference translation.

Output: Model-generated English translation of the Chinese source sentence.

Scoring recipe

def evaluate(predictions, references, human_scores):
    # Compute taxonomy-aware scores for each prediction
    model_scores = [compute_acre(pred, ref) for pred, ref in zip(predictions, references)]
    # Align with human judgment scores
    pearson_r = pearsonr(model_scores, human_scores)
    spearman_rho = spearmanr(model_scores, human_scores)
    return {
        'ACRE': mean(model_scores),
        'Pearson_r': pearson_r,
        'Spearman_rho': spearman_rho
    }

Common pitfalls

• Standard surface-level metrics like BLEU and COMET often yield high scores but fail to detect critical failures in preserving figurative or context-dependent cultural meaning.
• Evaluation requires span-level inspection; models must correctly translate the specific culture-loaded expression span rather than just achieving sentence-level fluency.
• Instances with weak contextual support or insufficient cultural salience are explicitly filtered out, so results may not generalize to ambiguous or low-context cultural terms.

Evidence (verbatim from paper)

Table 2. Pearson (r) and Spearman (ρ) correlation coefficients between metrics and human annotations on CulT-Eval. | Metric | Pearson $r$ | Spearman $

ho$ | | --- | --- | --- | | BLEU | 30.2 | 28.4 | | ChrF++ | 22.4 | 20.1 | | BERTScore | 27.5 | 25.3 | | COMET | 44.5 | 39.0 | | MetricX-QE | 24.6 | 22.8 | | ACRE (Ours) | 68.4 | 65.1 |

Citation

@misc{han2026culteval,
  title={From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation},
  author={Han et al. (2026)},
  year={2026},
  note={arXiv:2603.17303}
}

• arXiv: 2603.17303