Quantemp Eval

quantemp-eval

QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims — Venktesh V et al. (arXiv:2403.17169, 2024)

What this evaluates

Evaluates a model's ability to fact-check real-world numerical claims containing statistical and temporal expressions. It probes evidence retrieval, claim decomposition, and natural language inference to predict veracity (True, False, or Conflicting).

Datasets

• NumTemp — total 15514; splits: train (-1), val (-1), test (-1)

Metrics

• Macro-F1 (primary) — range: percent
  • Unweighted mean of the F1 scores for each of the three veracity classes (True, False, Conflicting). Calculated as the average of per-class precision-recall F1 scores.
• Weighted-F1 — range: percent
  • Weighted mean of the F1 scores for each veracity class, where weights correspond to the number of true instances for each class.

Input / output format

Input: Claim text (optionally decomposed into sub-questions or program steps) concatenated with retrieved evidence snippets (top 3) using separators.

Output: Veracity label: True, False, or Conflicting.

Scoring recipe

def compute_macro_f1(preds, gold):
    classes = ['True', 'False', 'Conflicting']
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• Evidence leakage: Using ground-truth justification paragraphs directly for retrieval instead of simulating a real-world retrieval pipeline.
• Training on mixed or non-numerical claims significantly degrades performance on numerical verification tasks.
• Ignoring claim decomposition leads to poor retrieval of quantitative evidence, hurting F1 scores.

Evidence (verbatim from paper)

Table 2: Results of different models on NumTemp (categorical and full) with Roberta-Large-MNLI as the NLI model. M-F1 : Macro-F1, W-F1 : Weighted-F1 and C-F1 refers to F1 score for Conflicting class.

Citation

@misc{venktesh2024quantemp,
  title={QuanTemp: A real-world open-domain benchmark for fact-checking numerical claims},
  author={Venktesh V et al.},
  year={2024},
  note={arXiv:2403.17169}
}

• arXiv: 2403.17169