Pubmedfact1k Eval

pubmedfact1k-eval

Uncertainty-Aware Web-Conditioned Scientific Fact-Checking — Vinod et al. (2026) (arXiv:2604.11036, 2026)

What this evaluates

This evaluation probes a model's ability to verify scientific claims in a three-way classification setting that includes uncertainty abstention. It measures performance on Supported, Refuted, and NEI (Not Enough Information) labels, testing the model's capacity to avoid overconfident predictions when evidence is insufficient or conflicting.

Datasets

• PubMedFact1k — total ?; splits: test (-1)

Metrics

• Macro F1 (primary) — range: percent
  • Unweighted mean of F1 scores computed independently for each class (Supported, Refuted, NEI).

Input / output format

Input: Scientific claim paired with context (local evidence snippets or retrieved web documents).

Output: A single verification label: 'Supported', 'Refuted', or 'NEI'.

Scoring recipe

def compute_pubmedfact_metrics(preds, golds):
    classes = ['Supported', 'Refuted', 'NEI']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(preds, golds) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(preds, golds) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Confusing the 2-way (Supported/Refuted) and 3-way (Supported/Refuted/NEI) evaluation regimes, as different datasets use different label sets.
• Reporting standard accuracy instead of Balanced Accuracy, which fails to account for class imbalance in datasets like CLIMATE-FEVER.
• Aggregating per-atom predictions using simple majority vote instead of the calibrated judge, which causes a ~14.6 F1 drop on BIONLI-300.

Evidence (verbatim from paper)

Table 2. Main results. Balanced Accuracy, Recall, and F1 on BIONLI-300; Macro-F1 on PubMedFact1k; and Balanced Accuracy, Recall (Supports), and F1 (Supports) on CLIMATE-FEVER, evaluated only on the Supported/Refuted subsets.

Citation

@misc{vinod2026uncertainty,
  title={Uncertainty-Aware Web-Conditioned Scientific Fact-Checking},
  author={Vinod et al. (2026)},
  year={2026},
  note={arXiv:2604.11036}
}

• arXiv: 2604.11036