M2 Verify Eval

m2-verify-eval

M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency — Ansari et al. (2026) (arXiv:2604.01306, 2026)

What this evaluates

Evaluates a model's ability to verify scientific claims by cross-referencing textual assertions with provided multimodal evidence (figures/diagrams). It probes cross-modal reasoning, spatial/anatomical understanding, and the generation of factually grounded explanations.

Datasets

• M2-Verify-Med — total ?; splits: test (-1)
• M2-Verify-Gen — total ?; splits: test (-1)

Metrics

• Macro-F1 (primary) — range: [0, 1]
  • The unweighted mean of recall or precision for each class. Calculated as the average of per-class F1 scores across all verification labels.
• Micro-F1 — range: [0, 1]
  • The global F1 score calculated by summing true positives, false positives, and false negatives across all classes before computing the metric.
• BLEU-2 — range: [0, 1]
  • N-gram precision metric measuring the overlap of 2-grams between generated explanations and gold references.
• ROUGE-L — range: [0, 1]
  • Recall-oriented metric based on the longest common subsequence between generated and reference explanations.
• METEOR — range: [0, 1]
  • N-gram metric that incorporates synonymy, stemming, and paraphrasing to measure alignment between generated and reference explanations.
• LLM-as-a-Judge — range: [0, 10]
  • Automated scoring of explanation quality across five dimensions: correctness, relevance, completeness, entailment, and clarity, using a dedicated LLM judge.
• Expert Faithfulness — range: [1, 5]
  • Human expert rating on a 1-5 scale assessing the logical faithfulness and factual accuracy of model-generated explanations.

Input / output format

Input: A textual scientific claim paired with a corresponding scientific figure or diagram (image).

Output: A verification label (e.g., support/refute) and a generated natural language explanation justifying the claim-evidence alignment.

Scoring recipe

def evaluate(predictions, gold):
    labels_pred = [p['label'] for p in predictions]
    labels_gold = [g['label'] for g in gold]
    macro_f1 = f1_score(labels_gold, labels_pred, average='macro')
    micro_f1 = f1_score(labels_gold, labels_pred, average='micro')
    exp_pred = [p['explanation'] for p in predictions]
    exp_gold = [g['explanation'] for g in gold]
    bleu2 = compute_bleu(exp_gold, exp_pred, n=2)
    rouge_l = compute_rouge(exp_gold, exp_pred, type='L')
    meteor = compute_meteor(exp_gold, exp_pred)
    return {'macro_f1': macro_f1, 'micro_f1': micro_f1, 'bleu2': bleu2, 'rouge_l': rouge_l, 'meteor': meteor}

Common pitfalls

• Models frequently exhibit 'visual bypassing', ignoring provided images and relying on parametric memory instead.
• Performance degrades sharply on high-complexity perturbations (e.g., anatomical shifts) compared to low-complexity ones, which can be misinterpreted as general capability failure rather than reasoning complexity limits.
• LLM-as-a-Judge scores for semantic validity often diverge from human expert faithfulness ratings, requiring careful calibration.

Evidence (verbatim from paper)

Following (Zhang and Lee, 2025) we report Macro F1 and Micro F1 scores for verification. For explanations we calculate N-gram metrics (BLEU-2, ROUGE-L, and METEOR) alongside an LLM-as-a-Judge measuring correctness, relevance, completeness, entailment, and clarity.

Citation

@misc{ansari2026m2verify,
  title={M2-Verify: A Large-Scale Multidomain Benchmark for Checking Multimodal Claim Consistency},
  author={Ansari et al. (2026)},
  year={2026},
  note={arXiv:2604.01306}
}

• arXiv: 2604.01306