Quan Temp Plus Eval

quan-temp-plus-eval

A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition — Venktesh et al. (2025) (arXiv:2510.22055, 2025)

What this evaluates

Evaluates open-domain numerical fact-checking by testing how well models can verify claims using decomposed queries and retrieved evidence. It probes the impact of claim decomposition quality on evidence retrieval and downstream NLI-based verification accuracy.

Datasets

• QuanTemp++ — total 15514; splits: train (9935), val (3084), test (2495); repo https://github.com/VenkteshV/QuanTemp_Plus

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted veracity labels (True, False, Contradiction) out of the total number of claims.
• macro-F1 (M-F1) — range: [0, 1]
  • Unweighted mean of F1 scores for each class (True, False, Contradiction), treating all classes equally.
• weighted-F1 (W-F1) — range: [0, 1]
  • F1 score averaged over classes, weighted by the number of true instances for each class.

Input / output format

Input: Claim text and top-k retrieved evidence snippets from the QuanTemp++ collection.

Output: Veracity label prediction: True, False, or Contradiction.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    # Per-class F1, macro-F1, and weighted-F1 are computed using standard classification metrics
    # e.g., sklearn.metrics.f1_score(gold_labels, predictions, average='macro')
    return {'accuracy': accuracy, 'macro_f1': macro_f1, 'weighted_f1': weighted_f1}

Common pitfalls

• The evaluation uses a fixed top-k retrieval (k=3) for the main NLI verification step, so models must handle evidence ordering and potential noise in retrieved snippets rather than performing end-to-end retrieval from scratch.
• Per-class F1 distinguishes between True, False, and Contradiction, which differs from standard binary fact-checking benchmarks and requires careful handling of the NLI label space.
• Evidence collection is pre-computed via FCDecomp; the benchmark evaluates verification given these snippets, not the decomposition or retrieval process itself.

Evidence (verbatim from paper)

We employ Contriever as retrieval model afetr comparing several approaches like BM25, ANCE, Tas-b on validation set. We experiment with different values of k=1,3,5,7,10 for top-k evidence retrieval and observe k=3 to provide best NLI performance on validation set. For each of these settings, we evaluate the accuracy, per-class F1, macro-F1 (M-F1), and weighted-F1 (W-F1) scores.

Citation

@misc{venktesh2025quantemp,
  title={A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition},
  author={Venktesh et al. (2025)},
  year={2025},
  note={arXiv:2510.22055}
}

• arXiv: 2510.22055