# Sciclaimeval Eval

> Evaluates multimodal models' ability to verify scientific claims by classifying them as Supported or Refuted based on cross-modal evidence (tables or figures). It probes visual reasoning, table parsing, and resistance to dataset biases or superficial shortcuts. Use when the user wants to benchmark on SciClaimEval, or asks about evaluating this task. Reports macro-F1.

- Skill: `qhjqhj00/sciclaimeval-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/sciclaimeval-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/sciclaimeval-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/sciclaimeval-eval

---


# sciclaimeval-eval

> SciClaimEval: Cross-modal Claim Verification in Scientific Papers — Ho et al. (2026) (arXiv:2602.07621, 2026)

## What this evaluates

Evaluates multimodal models' ability to verify scientific claims by classifying them as Supported or Refuted based on cross-modal evidence (tables or figures). It probes visual reasoning, table parsing, and resistance to dataset biases or superficial shortcuts.

## Datasets

- **SciClaimEval** — total 1664; splits: val (747), test (917)

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Standard macro-averaged F1 score for binary classification (Supported vs. Refuted). Computed as the unweighted mean of per-class F1 scores.
- `Pair Accuracy` — range: [0, 1]
  - Number of correctly predicted pairs divided by the total number of pairs. A pair is correct only if both the Supported and Refuted samples for the same claim are predicted correctly.

## Input / output format

**Input**: A scientific claim text, a visual evidence item (table image or figure), and optionally preceding context sentences.

**Output**: Binary classification label: 'Supported' or 'Refuted'.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, pairs):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold_labels):
        if p == g == 'Supported': tp += 1
        elif p == 'Supported' and g == 'Refuted': fp += 1
        elif p == 'Refuted' and g == 'Supported': fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    correct_pairs = sum(1 for pair in pairs if all(predictions[i] == gold_labels[i] for i in pair))
    pair_acc = correct_pairs / len(pairs)
    return {'macro-F1': f1, 'Pair Accuracy': pair_acc}
```

## Common pitfalls

- Relying on dataset biases or reasoning shortcuts can inflate macro-F1 scores without indicating genuine evidence understanding.
- Pair Accuracy has a random baseline of 0.25 (vs 0.5 for macro-F1), making it significantly stricter and more discriminative.
- Providing extra context sentences can sometimes hurt performance compared to a no-context setup, as joint reasoning over context and evidence increases difficulty.

## Evidence (verbatim from paper)

> Following previous work on the scientific claim verification task, we use the macro-F1 evaluation metric in our experiments. However, since the task is a binary classification problem (Supported vs. Refuted), macro-F1 alone may be insufficient. A model may achieve a reasonable macro-F1 score through lucky guesses or by exploiting reasoning shortcuts or dataset biases. To mitigate this effect, we introduce a new evaluation metric, Pair Accuracy, defined as the number of correctly predicted pairs divided by the total number of pairs. A pair is considered correct only if both samples associated with the same claim, one Supported and one Refuted, are predicted correctly.

## Citation

```bibtex
@misc{ho2026sciclaimeval,
  title={SciClaimEval: Cross-modal Claim Verification in Scientific Papers},
  author={Ho et al. (2026)},
  year={2026},
  note={arXiv:2602.07621}
}
```

- arXiv: 2602.07621