Oecd Tabular Fact Checking Eval

oecd-tabular-fact-checking-eval

Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data — Devasier et al. (2026) (arXiv:2601.17232, 2026)

What this evaluates

Evaluates an LLM's ability to verify factual claims against high-volume tabular data by measuring evidence retrieval (table and data level) and claim factuality classification. It also probes whether models rely on parametric knowledge versus active retrieval and reasoning over structured data.

Datasets

• OECD Tabular Fact-Checking Dataset — total 78503; splits: test (-1)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly predicted verdicts (True, False, or NEI) or correctly retrieved tables/data rows out of the total number of instances. Calculated as (Correct Predictions / Total Instances) * 100.
• Precision, Recall, F1-score — range: [0, 1]
  • Standard classification metrics for claim-knowledge alignment, measuring the model's ability to predict True/False based on internal knowledge versus the generated claims.
• Human quality rate — range: percent
  • Percentage of randomly sampled claims manually annotated as 'good' by native speakers.

Input / output format

Input: A synthetic factual claim derived from a high-volume tabular dataset, accompanied by the corresponding table(s) or database context for evidence retrieval.

Output: A predicted verdict (True, False, or NEI), optionally accompanied by retrieved evidence (specific table or data rows) or a generated query.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_retrieval_accuracy(retrieved_ids, gold_ids):
    correct = sum(1 for r, g in zip(retrieved_ids, gold_ids) if r == g)
    return (correct / len(gold_ids)) * 100

Common pitfalls

• LLM judges struggle with exact numeric values and may classify claims as False due to minor rounding differences unless explicitly prompted to allow tolerance.
• High rates of 'Not Enough Information' (NEI) predictions can skew overall accuracy metrics if not analyzed separately from True/False verdicts.
• Human annotation quality varies significantly across languages, with lower quality rates observed for non-English languages like Hindi.

Evidence (verbatim from paper)

Our baseline performed significantly better on both predicted and gold tables, with the thinking-enabled LLM consistently performing the best, with 36% and 20.4% accuracy on the gold and predicted tables, respectively.

Citation

@misc{devasier2026frame,
  title={Frame-Guided Synthetic Claim Generation for Automatic Fact-Checking Using High-Volume Tabular Data},
  author={Devasier et al. (2026)},
  year={2026},
  note={arXiv:2601.17232}
}

• arXiv: 2601.17232