Morphogen Eval

morphogen-eval

MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation — Aggarwal et al. (2026) (arXiv:2604.18914, 2026)

What this evaluates

This benchmark evaluates a model's ability to perform gender-aware morphological generation by rewriting first-person sentences in French, Arabic, and Hindi to the opposite grammatical gender. It probes compositional morphosyntactic reasoning, testing whether models can correctly transform gendered terms while preserving semantic meaning and grammatical structure across bidirectional transformations.

Datasets

• MORPHOGEN — total ?; splits: test (-1); repo https://github.com/arnav10goel/Morphogen

Metrics

• Sentence-Level Gender Accuracy (SGA) — range: percent
  • Average across N sentences of the ratio of correctly modified gendered terms to total gendered terms per sentence: (1/N) Σ (|Gendered_i ∩ Mismatch_i^c| / |Gendered_i|). Captures sentence-level precision.
• Gender IoU Score (GIoU) (primary) — range: percent
  • Average across N sentences of the ratio of correctly transformed gendered terms to the union of gendered and mismatched terms: (1/N) Σ (|Gendered_i ∩ Mismatch_i^c| / |Gendered_i ∪ Mismatch_i|). Penalizes both over-generation and under-generation.
• Corpus-Level Gender Accuracy (CGA) — range: percent
  • Ratio of correctly generated gendered terms across the entire test set to the total number of reference gendered terms: Σ|Gendered_i ∩ Mismatch_i^c| / Σ|Gendered_i|. Provides a holistic word-level measure.

Input / output format

Input: A first-person sentence in French, Arabic, or Hindi containing gendered terms that must be transformed to the opposite grammatical gender.

Output: A rewritten sentence where all gendered terms are correctly transformed to the opposite gender, preserving original meaning and grammatical structure.

Scoring recipe

# Pseudo-code for SGA, GIoU, CGA
N = len(test_set)
total_sga = 0
total_giou = 0
total_correct = 0
total_gold = 0

for sentence in test_set:
    gold_gendered = extract_gendered_terms(sentence.gold)
    pred_gendered = extract_gendered_terms(sentence.pred)
    correct = gold_gendered.intersection(pred_gendered)
    
    total_sga += len(correct) / len(gold_gendered)
    total_giou += len(correct) / len(gold_gendered.union(pred_gendered))
    total_correct += len(correct)
    total_gold += len(gold_gendered)

sga = (total_sga / N) * 100
giou = (total_giou / N) * 100
cga = (total_correct / total_gold) * 100

Common pitfalls

• Models frequently suffer from gender interference, incorrectly altering the gender of words associated with entities other than the explicit speaker.
• Performance gaps often exist between masculine-to-feminine and feminine-to-masculine conversions, requiring disaggregated reporting (ΔSGA) rather than relying solely on aggregate scores.
• Over-generation (modifying non-gendered terms) and under-generation (failing to modify required terms) are both penalized by GIoU, making it stricter than simple accuracy and sensitive to hallucination.

Evidence (verbatim from paper)

To evaluate model performance on the MORPHOGEN benchmark, we propose three complementary metrics that measure an LLM’s ability to correctly perform gender-aware morphological transformations at different granularities. ... The proposed metrics are defined as follows: (1) Sentence-Level Gender Accuracy (SGA): This metric measures the proportion of correctly generated gendered terms in each sentence. ... (2) Gender IoU Score (GIoU): Inspired by the Intersection-over-Union (IoU) metric commonly used in object detection, GIoU metric provides a stricter and more comprehensive measure of morphological transformation quality. It penalizes both over-generation (modifying non-gendered terms or incorrect gendered entities) and under-generation (failing to modify gendered terms). ... (3) Corpus-Level Gender Accuracy (CGA) This is a corpus-level aggregation of gender correctness.

Citation

@misc{aggarwal2026morphogen,
  title={MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation},
  author={Aggarwal et al. (2026)},
  year={2026},
  note={arXiv:2604.18914}
}

• arXiv: 2604.18914