Musciclaims Eval

musciclaims-eval

MuSciClaims: Multimodal Scientific Claim Verification — Lal et al. (2025) (arXiv:2506.04585, 2025)

What this evaluates

Multimodal scientific claim verification, requiring models to read complex figures and captions to determine if a scientific claim is supported, neutral, or contradicted. It also probes evidence localization, basic visual understanding, cross-modal aggregation, and epistemic sensitivity.

Datasets

• MuSciClaims — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Computed per class and macro-averaged across Support, Neutral, and Contradict.
• Exact Match — range: [0, 1]
  • Binary metric returning 1 if the model's one-word answer exactly matches the gold answer, else 0.

Input / output format

Input: Multi-panel figure, figure caption, and a scientific claim.

Output: For ClaimVerification: a prediction of 'Support', 'Neutral', or 'Contradict'. For EvidenceLocalization: relevant panel names plus the prediction. For BasicVisualUnderstanding: a single-word answer.

Scoring recipe

def compute_f1(predictions, golds):
    tp = fp = fn = 0
    for p, g in zip(predictions, golds):
        if p == g: tp += 1
        elif p != g: fp += 1; fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_em(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

Common pitfalls

• Models exhibit a strong bias toward predicting 'Support' regardless of the actual evidence.
• Models frequently fail to correctly localize evidence to specific figure panels or aggregate cross-modal information.
• Models show low epistemic sensitivity, often failing to change predictions consistently when claims are perturbed.

Evidence (verbatim from paper)

MuSciClaims is designed as a ClaimVerification task. Each data point contains a claim, an associated (multi-panel) figure (and caption) and a label (Support, Neutral, Contradict). Given the figure (and caption) and a claim, models must generate a prediction about whether the claim is supported. We evaluate models on this task using standard metrics of precision, recall and F1 score.

Citation

@misc{lal2025musciclaims,
  title={MuSciClaims: Multimodal Scientific Claim Verification},
  author={Lal et al. (2025)},
  year={2025},
  note={arXiv:2506.04585}
}

• arXiv: 2506.04585