Prismm Bench Eval

prismm-bench-eval

PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies — Selch et al. (2025) (arXiv:2510.16505, 2025)

What this evaluates

Evaluates large multimodal models' ability to detect, correct, and reason over real-world multimodal inconsistencies in scientific papers. It probes inter-modal mismatch detection, structured reasoning, and robustness to linguistic shortcuts versus genuine visual grounding.

Datasets

• PRISMM-Bench — total 262; splits: test (262); repo https://github.com/da-luggas/prismm-bench

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly predicted options out of the total number of instances. Computed as (correct predictions / total instances) * 100.
• Visual Reliance Ratio (R) — range: percent
  • Measures dependence on visual context versus linguistic priors. Formula: R = (Acc_with_context - Acc_without_context) / (1 - Acc_without_context). Higher R indicates stronger reliance on visual grounding.

Input / output format

Input: Multimodal scientific documents containing text, figures, tables, and equations. Inputs are provided at three contextual granularity levels: Focused (cropped images/text excerpts), Page, or Document (full PDF). Each instance presents a multiple-choice question to identify inconsistencies, propose remedies, or match pairs.

Output: Strictly the letter corresponding to the chosen option. For reasoning-enabled models, the final answer is extracted from within ... tags before postprocessing.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

def compute_visual_reliance_ratio(acc_with_context, acc_without_context):
    return (acc_with_context - acc_without_context) / (1 - acc_without_context)

Common pitfalls

• Models can exploit linguistic regularities in natural language multiple-choice options, achieving inflated accuracy without genuine multimodal reasoning.
• Context granularity significantly impacts performance; models degrade toward random chance when moving from Focused to Document context due to distraction and long-range grounding challenges.
• Reasoning traces must be explicitly parsed (e.g., extracting content between tags) rather than assuming the final answer is always at the end of the generation.

Evidence (verbatim from paper)

Accuracy (%) of 21 diverse LMMs across three tasks (Ident, Remedy, Match) and three levels of contextual granularity (Sec.[3.4]). To quantify how much models and human rely on visual evidence versus linguistic priors, we compute the Visual Reliance Ratio R, adapted from the normalized Perceptual Score (Gat et al., [2021]): R = (Acc_with_context - Acc_without_context) / (1 - Acc_without_context)

Citation

@misc{selch2025prismmbench,
  title={PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies},
  author={Selch et al. (2025)},
  year={2025},
  note={arXiv:2510.16505}
}

• arXiv: 2510.16505