Mmjee Eval Eval

mmjee-eval-eval

mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models — Arka Mukherjee et al. (2025) (arXiv:2511.09339, 2025)

What this evaluates

Evaluates scientific reasoning in vision-language models using bilingual (English/Hindi) multimodal questions from India's JEE Advanced exam. It probes cross-domain concept integration, meta-cognitive self-correction, and cross-lingual consistency under exam-style constraints.

Datasets

• mmJEE-Eval — total 1460; splits: 2019-2024 (1270), 2025 held-out (190); repo https://github.com/ArkaMukherjee0/mmJEE-Eval

Metrics

• Pass@1 accuracy (primary) — range: percent
  • Accuracy calculated as the fraction of correctly answered questions, averaged over k=10 independent inference runs per instance.

Input / output format

Input: Multimodal exam questions containing text and images/diagrams, presented in either English or Hindi.

Output: Multiple-choice answers (single-correct or multiple-correct MCQs), or structured reasoning chains for error detection and correction steps.

Scoring recipe

def compute_pass_at_1(predictions_per_run, gold_labels, k=10):
    run_accuracies = []
    for run_preds in predictions_per_run:
        correct = sum(1 for p, g in zip(run_preds, gold_labels) if p == g)
        run_accuracies.append(correct / len(gold_labels))
    return sum(run_accuracies) / k

Common pitfalls

• Confusing raw Pass@1 accuracy with confidence-thresholded scores (Marks w/ CT), which apply self-consistency and selective answering to avoid negative marking.
• Assuming error detection/correction (EP/EC) chaining yields similar gains to Pass@k sampling; the protocol shows EP/EC improves scores by only ~2-6% compared to ~30% for Pass@3 due to metacognitive limitations.
• Overlooking language-specific failure asymmetries (e.g., English-dominant training) when evaluating cross-lingual consistency, as models perform significantly worse on Hindi-only correct cases.

Evidence (verbatim from paper)

Table 3 presents the Pass@1 accuracy averaged over k=10 runs (Appendix[B] justifies this choice) of all evaluated models across subjects and question types (RQ1). Our motivation for selecting Pass@1 is two-fold: (a) OpenAI Simple Evals promotes it, and (b) allows fair comparison with other industry standard vision benchmarks.

Citation

@misc{mukherjee2025mmjeeeval,
  title={mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models},
  author={Arka Mukherjee et al. (2025)},
  year={2025},
  note={arXiv:2511.09339}
}

• arXiv: 2511.09339