Meena Persianmmmu Eval

meena-persianmmmu-eval

MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment — Ghahroodi et al. (2025) (arXiv:2508.17290, 2025)

What this evaluates

This benchmark evaluates vision-language models on multimodal educational exam questions in Persian and English. It specifically probes visual grounding, reasoning capabilities, and robustness to missing or mismatched visual cues across different prompting strategies.

Datasets

• MEENA (PersianMMMU) — total 10500; splits: test (10500)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy computed as the fraction of questions where the model's extracted selected option matches the ground truth label.

Input / output format

Input: Text question ($q_*$) in Persian or English, optionally accompanied by an image ($x_*$), and up to four few-shot example triplets for the In-Context Learning setting. Questions are multiple-choice.

Output: Free-form text response from which a single option (letter or number) must be extracted. The evaluation pipeline uses regex or an LLM-as-a-judge to parse the final selected option.

Scoring recipe

def compute_accuracy(extracted_options, gold_options):
    correct = 0
    for pred, gold in zip(extracted_options, gold_options):
        if pred == gold:
            correct += 1
    return correct / len(gold_options)

Common pitfalls

• Regex-based option extraction fails in ~50% of cases, necessitating an LLM-as-a-judge that may introduce parsing bias or misinterpret non-standard model outputs.
• Models may achieve high accuracy in 'Without Image' or 'Wrong Image' settings by relying on textual priors rather than genuine visual grounding, masking multimodal deficiencies.
• Cross-linguistic performance gaps may reflect translation artifacts or native-language training data imbalances rather than true multimodal reasoning differences.

Evidence (verbatim from paper)

To assess model performance, it is essential to identify the option selected by the model in its generated response and use it to compute accuracy.

Citation

@misc{ghahroodi2025meena,
  title={MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment},
  author={Ghahroodi et al. (2025)},
  year={2025},
  note={arXiv:2508.17290}
}

• arXiv: 2508.17290