Histoires Morales Eval

histoires-morales-eval

Histoires Morales: A French Dataset for Assessing Moral Alignment — Leteno et al. (2025) (arXiv:2501.17117, 2025)

What this evaluates

Evaluates the linguistic quality and cultural appropriateness of the Histoires Morales dataset. It measures reference-free translation accuracy and assesses whether moral norms and actions align with native French speakers' cultural values.

Datasets

• Histoires Morales — total 12000; splits: full (12000); repo https://github.com/upunaprosk/histoires-morales

Metrics

• CometKiwi22 (primary) — range: [0, 1]
  • Reference-free quality estimation metric for sentence-level translation. Outputs a continuous score between 0 and 1, where higher values indicate better translation quality.
• cultural_alignment_agreement — range: percent
  • Categorical label derived from 4 native French annotators. Agreement if fewer than 2 annotators disagree, Disagreement if more than 2 disagree, Uncertainty if exactly 2 disagree.

Input / output format

Input: Translated French sentences categorized into Norm, Situation, Intention, Moral action, Moral consequence, Immoral action, or Immoral consequence.

Output: For translation: a float score between 0 and 1. For cultural alignment: a categorical label (Agreement, Disagreement, or Uncertainty).

Scoring recipe

def compute_cometkiwi22(predictions, gold):
    return cometkiwi22_model.predict(predictions, gold)  # Returns float in [0, 1]

def compute_cultural_agreement(predictions, gold):
    disagree_count = sum(1 for p in predictions if p == 'disagree')
    if disagree_count < 2:
        return 'Agreement'
    elif disagree_count > 2:
        return 'Disagreement'
    else:
        return 'Uncertainty'

Common pitfalls

• Reference-free QE models like CometKiwi22 may penalize context-sensitive phrasal verbs or collocations that are actually correct and culturally appropriate translations.
• Cultural alignment assessment relies on a small sample (500 items) and subjective annotator thresholds, which may not generalize to broader French-speaking populations or nuanced moral situations.

Evidence (verbatim from paper)

We measure the quality of translation with the CometKiwi22 reference-free quality estimation (QE) metric introduced by Rei et al., [2022]. This metric is suitable for sentence and word-level QE and supports English-to-French translations, with values between 0 and 1, and higher values indicating better translations.

Citation

@misc{leteno2025histoiresmorales,
  title={Histoires Morales: A French Dataset for Assessing Moral Alignment},
  author={Leteno et al. (2025)},
  year={2025},
  note={arXiv:2501.17117}
}

• arXiv: 2501.17117