Mme Sci Eval

mme-sci-eval

MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models — Ruan et al. (2025) (arXiv:2508.13938, 2025)

What this evaluates

Evaluates multimodal large language models on scientific reasoning across four disciplines (math, physics, chemistry, biology) and five languages. It probes cross-lingual consistency, modality robustness (text-only vs. image-only vs. image-text), and fine-grained domain knowledge under varying visual complexity.

Datasets

• MME-SCI — total 1019; splits: test (-1); repo https://github.com/JCruan519/MME-SCI

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Correctness is determined by an LLM-as-a-Judge comparing the model's generated response against the ground-truth answer using language-specific evaluation templates.

Input / output format

Input: A scientific question in one of five languages, optionally accompanied by an image (e.g., screenshot of a diagram or problem). The evaluation covers three modalities: text-only, image-only, and image-text.

Output: A natural language response answering the scientific question.

Scoring recipe

judge = LLM(temperature=0, max_tokens=8192)
correct = 0
for pred, gold, lang in zip(predictions, gold_answers, languages):
    prompt = build_judge_template(pred, gold, lang)
    verdict = judge.generate(prompt)
    if is_correct(verdict): correct += 1
accuracy = (correct / len(predictions)) * 100

Common pitfalls

• Image-only evaluation uses screenshot-based inputs that heavily penalize models with weak OCR or vision-language alignment, causing disproportionate accuracy drops for open-source models.
• LLM-as-a-Judge scoring requires language-specific templates; using a single English template across multilingual data introduces bias and inflates or deflates accuracy unfairly.
• Performance varies drastically by domain (e.g., physics vs. chemistry) and reasoning capability, so reporting only aggregate accuracy masks critical modality or subject-specific failures.

Evidence (verbatim from paper)

We adopt the ‘LLM-as-a-Judge’ paradigm (Zheng et al. [2023]; Gu et al. [2024]) and introduce different evaluation templates for various languages to assess the models. Unless specified otherwise, we configured the maximum number of new tokens to 8,192, and the temperature was set to 0. Compared to advanced closed-source models, the most powerful open-source models (Large group) demonstrates an average accuracy reduction of 13.94% across six scenarios.

Citation

@misc{ruan2025mme_sci,
  title={MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models},
  author={Ruan et al. (2025)},
  year={2025},
  note={arXiv:2508.13938}
}

• arXiv: 2508.13938