Mt Geneval Eval

mt-geneval-eval

MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation — Currey et al. (2022) (arXiv:2211.01355, 2022)

What this evaluates

Evaluates machine translation models' ability to correctly translate gender-specific words and maintain gender agreement across multiple languages. It also measures representational bias by comparing translation quality between male and female counterfactual sentence pairs.

Datasets

• MT-GenEval — total ?; splits: test (-1), contextual (-1), counterfactual (-1); repo https://github.com/amazon-research/machine-translation-gender-eval

Metrics

• accuracy (primary) — range: [0, 1]
  • Identify words in the contrastive reference that are not in the correct reference (unique_con = w_con \ w_ref). A segment is marked incorrect if the hypothesis contains any word from unique_con; otherwise it is correct. Accuracy is the fraction of correct segments.
• Δ_qual — range: other
  • Δ_qual = BLEU_male - BLEU_female, where BLEU_gender is the standard BLEU score computed on the subset of the counterfactual test set corresponding to that gender. Measures representational bias.

Input / output format

Input: Source sentence in a source language (e.g., English) to be translated into a target language with gendered morphology.

Output: Translated hypothesis sentence in the target language.

Scoring recipe

# accuracy
unique_con = set(w_con) - set(w_ref)
is_correct = len(unique_con.intersection(set(w_hyp))) == 0
accuracy = sum(is_correct) / len(segments)

# Δ_qual
bleu_male = compute_bleu(hypotheses_male, references_male)
bleu_female = compute_bleu(hypotheses_female, references_female)
delta_qual = bleu_male - bleu_female

Common pitfalls

• The accuracy metric relies on exact word overlap with contrastive references, so tokenization differences or morphological variations in the hypothesis can cause false negatives.
• On the counterfactual subset, a pair is only counted as correct if BOTH the original and its counterfactual counterpart are marked correct, which penalizes inconsistent models.
• The quality gap uses standard BLEU, which may not reflect gender-specific errors if the overall translation fluency is high.

Evidence (verbatim from paper)

We define accuracy of gender in translation on our test set as follows. Let $w_{hyp}$, $w_{ref}$, and $w_{con}$ denote the set of words in the hypothesis, reference, and contrastive reference, respectively. First, we obtain the set of words in the contrastive reference that are not in the correct reference: $unique_{con} = w_{con} \backslash w_{ref}$. We consider a segment incorrect if: $unique_{con} \cap w_{hyp} \neq \emptyset$. ... We define $\Delta_{\text{qual}}$ as: $\Delta_{\text{qual}} = \mathrm{BLEU}{\mathrm{male}} - \mathrm{BLEU}{\mathrm{female}}$ where $\mathrm{BLEU}_{\mathrm{gender}}$ is the BLEU score of the gender subset of the counterfactual test set.

Citation

@misc{currey2022mtgeneval,
  title={MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation},
  author={Currey et al. (2022)},
  year={2022},
  note={arXiv:2211.01355}
}

• arXiv: 2211.01355