Csmbench Eval

csmbench-eval

Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science — Zheng et al. (2026) (arXiv:2603.19327, 2026)

What this evaluates

Evaluates large multimodal models' ability to perceive, interpret, and reason about scientific figures across four hierarchical physical scales (atomic, micro, meso, macro) in materials science. It probes both discriminative visual matching and open-ended scientific narrative generation.

Datasets

• CSMBench — total 1041; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly selected multiple-choice options out of the total number of instances.
• BERTScore F1 — range: [0, 1]
  • F1 score computed from contextual embeddings to measure semantic similarity between generated text and ground truth.
• STS — range: [-1, 1]
  • Cosine similarity between 384-dimensional sentence-transformer vectors of the model output and ground truth.
• LLM-as-a-Judge score — range: [1, 10]
  • Score from 1 to 10 assigned by GPT-4o-2024-11-20 to semantically verify answer correctness against predictions.

Input / output format

Input: A materials science figure paired with a text prompt. Multiple-choice tasks provide candidate captions; open-ended tasks request a description or explanation.

Output: Multiple-choice: A single selected caption/option. Open-ended: A free-form textual description or scientific narrative.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'multiple_choice':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif task == 'open_ended':
        bert_f1 = bertscore_f1(predictions, golds)
        sts = cosine_similarity(st_embed(predictions), st_embed(golds))
        llm_scores = [gpt4o_judge(p, g) for p, g in zip(predictions, golds)]
        return {'BERTScore F1': bert_f1, 'STS': np.mean(sts), 'LLM-as-a-Judge': np.mean(llm_scores)}

Common pitfalls

• Models show scale-dependent performance drops on atomic/macro scales due to heterogeneous schematic diagrams compared to standardized micro/meso images.
• LLM-as-a-Judge scores may exhibit family bias (e.g., GPT-4o favoring GPT-family models), requiring cross-validation with STS.
• Increasing model parameters does not guarantee better performance on complex scientific reasoning tasks; architectural design matters more.

Evidence (verbatim from paper)

For the multiple-choice QA task, we present accuracy as the primary metric. For the open-ended QA task, we report three metrics. For BERTScore, we employ the F1 score to evaluate semantic similarity using contextual embeddings. Semantic Textual Similarity (STS) is used to provide a high-level conceptual assessment of the overall interpretation. We calculate STS by projecting both the model output and ground truth into a unified 384-dimensional vector space using a sentence-transformer model, and computing their cosine similarity. For the LLM-as-a-Judge score, we use GPT-4o-2024-11-20 as the judge to semantically verify the correctness of answers against model predictions... where the score is scaled from 1 to 10.

Citation

@misc{zheng2026csmbench,
  title={Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2603.19327}
}

• arXiv: 2603.19327