Climate Fever Eval

climate-fever-eval

Uncertainty-Aware Web-Conditioned Scientific Fact-Checking — Vinod et al. (2026) (arXiv:2604.11036, 2026)

What this evaluates

This evaluation probes a model's ability to verify scientific claims in a binary classification setting, specifically testing out-of-domain generalization. It measures performance on Supported vs. Refuted labels, emphasizing robustness when applied to climate-related claims outside the training distribution.

Datasets

• CLIMATE-FEVER — total ?; splits: test (-1)

Metrics

• Balanced Accuracy (primary) — range: percent
  • Average of recall obtained on each class: (TPR + TNR) / 2. Computed for the Supported and Refuted classes.
• F1 (Supports) — range: percent
  • Harmonic mean of precision and recall for the Supported class: 2 * (Precision * Recall) / (Precision + Recall).
• Recall (Supports) — range: percent
  • True positive rate for the Supported class: TP / (TP + FN).

Input / output format

Input: Scientific claim paired with context (local evidence snippets or retrieved web documents).

Output: A single verification label: 'Supported' or 'Refuted'.

Scoring recipe

def compute_climate_fever_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 'Supported' and g == 'Supported')
    fn = sum(1 for p, g in zip(preds, golds) if p == 'Refuted' and g == 'Supported')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'Supported' and g == 'Refuted')
    tn = sum(1 for p, g in zip(preds, golds) if p == 'Refuted' and g == 'Refuted')
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0
    bal_acc = (tpr + tnr) / 2
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * (prec * tpr) / (prec + tpr) if (prec + tpr) > 0 else 0
    return bal_acc, f1, tpr

Common pitfalls

• Confusing the 2-way (Supported/Refuted) and 3-way (Supported/Refuted/NEI) evaluation regimes, as different datasets use different label sets.
• Reporting standard accuracy instead of Balanced Accuracy, which fails to account for class imbalance in datasets like CLIMATE-FEVER.
• Aggregating per-atom predictions using simple majority vote instead of the calibrated judge, which causes a ~14.6 F1 drop on BIONLI-300.

Evidence (verbatim from paper)

Table 2. Main results. Balanced Accuracy, Recall, and F1 on BIONLI-300; Macro-F1 on PubMedFact1k; and Balanced Accuracy, Recall (Supports), and F1 (Supports) on CLIMATE-FEVER, evaluated only on the Supported/Refuted subsets.

Citation

@misc{vinod2026uncertainty,
  title={Uncertainty-Aware Web-Conditioned Scientific Fact-Checking},
  author={Vinod et al. (2026)},
  year={2026},
  note={arXiv:2604.11036}
}

• arXiv: 2604.11036