Factir Eval

factir-eval

FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking — Venktesh et al. (2025) (arXiv:2502.06006, 2025)

What this evaluates

Evaluates open-domain retrieval and re-ranking systems on real-world fact-checking claims. It probes the ability to retrieve indirect, multifaceted evidence from unstructured web sources to support or refute complex queries involving health, politics, and economics.

Datasets

• FactIR — total ?; splits: test (-1); repo https://github.com/factiverse/factIR

Metrics

• nDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff k. Computed as DCG@k divided by the ideal DCG@k, where DCG discounts relevance by the logarithm of the document's rank position.
• Recall@k — range: [0, 1]
  • The proportion of relevant ground-truth documents retrieved within the top-k ranked results.

Input / output format

Input: A fact-checking claim (served as the query) and a large corpus of unstructured web documents.

Output: A ranked list of retrieved documents (or top-k document IDs) for each claim.

Scoring recipe

def compute_metrics(predictions, gold, k):
    recall_k = len(set(predictions[:k]) & gold) / len(gold)
    dcg = sum(1 / math.log2(i + 2) for i, doc in enumerate(predictions[:k]) if doc in gold)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(gold), k)))
    ndcg_k = dcg / idcg if idcg > 0 else 0.0
    return ndcg_k, recall_k

Common pitfalls

• Queries are real-world production claims requiring indirect reasoning, not simple keyword searches.
• Relevance is multi-faceted (partial relevance, stance), so binary relevance assumptions can mislead evaluation.
• The cutoff k is not fixed in the protocol; results must explicitly state the k value used.

Evidence (verbatim from paper)

Metrics: We choose Normalized Discounted Cumulative Gain (nDCG@k) and Recall@k as the primary metrics for our results.

Citation

@misc{venktesh2025factir,
  title={FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking},
  author={Venktesh et al. (2025)},
  year={2025},
  note={arXiv:2502.06006}
}

• arXiv: 2502.06006