Sciclaimeval Eval

sciclaimeval-eval

SciClaimEval: Cross-modal Claim Verification in Scientific Papers — Ho et al. (2026) (arXiv:2602.07621, 2026)

What this evaluates

Evaluates multimodal models' ability to verify scientific claims by classifying them as Supported or Refuted based on cross-modal evidence (tables or figures). It probes visual reasoning, table parsing, and resistance to dataset biases or superficial shortcuts.

Datasets

• SciClaimEval — total 1664; splits: val (747), test (917)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Standard macro-averaged F1 score for binary classification (Supported vs. Refuted). Computed as the unweighted mean of per-class F1 scores.
• Pair Accuracy — range: [0, 1]
  • Number of correctly predicted pairs divided by the total number of pairs. A pair is correct only if both the Supported and Refuted samples for the same claim are predicted correctly.

Input / output format

Input: A scientific claim text, a visual evidence item (table image or figure), and optionally preceding context sentences.

Output: Binary classification label: 'Supported' or 'Refuted'.

Scoring recipe

def compute_metrics(predictions, gold_labels, pairs):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold_labels):
        if p == g == 'Supported': tp += 1
        elif p == 'Supported' and g == 'Refuted': fp += 1
        elif p == 'Refuted' and g == 'Supported': fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    correct_pairs = sum(1 for pair in pairs if all(predictions[i] == gold_labels[i] for i in pair))
    pair_acc = correct_pairs / len(pairs)
    return {'macro-F1': f1, 'Pair Accuracy': pair_acc}

Common pitfalls

• Relying on dataset biases or reasoning shortcuts can inflate macro-F1 scores without indicating genuine evidence understanding.
• Pair Accuracy has a random baseline of 0.25 (vs 0.5 for macro-F1), making it significantly stricter and more discriminative.
• Providing extra context sentences can sometimes hurt performance compared to a no-context setup, as joint reasoning over context and evidence increases difficulty.

Evidence (verbatim from paper)

Following previous work on the scientific claim verification task, we use the macro-F1 evaluation metric in our experiments. However, since the task is a binary classification problem (Supported vs. Refuted), macro-F1 alone may be insufficient. A model may achieve a reasonable macro-F1 score through lucky guesses or by exploiting reasoning shortcuts or dataset biases. To mitigate this effect, we introduce a new evaluation metric, Pair Accuracy, defined as the number of correctly predicted pairs divided by the total number of pairs. A pair is considered correct only if both samples associated with the same claim, one Supported and one Refuted, are predicted correctly.

Citation

@misc{ho2026sciclaimeval,
  title={SciClaimEval: Cross-modal Claim Verification in Scientific Papers},
  author={Ho et al. (2026)},
  year={2026},
  note={arXiv:2602.07621}
}

• arXiv: 2602.07621