Fact Checking Arena Eval

fact-checking-arena-eval

Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking — Lin et al. (2026) (arXiv:2601.02669, 2026)

What this evaluates

Evaluates LLMs on multi-hop fact-checking by measuring claim extraction, evidence retrieval, and justification quality. Uses an arena-style pairwise comparison framework with LLM judges to rank models across multiple reasoning dimensions.

Datasets

• HOVER — total ?; splits: test (200)
• FEVERIOUS — total ?; splits: test (200)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly verified claims out of the total evaluated claims. Calculated as (correct predictions / total claims) * 100.
• Overall Judge — range: other
  • Aggregate score derived from pairwise arena-style judgments across multiple dimensions including claim extraction, evidence retrieval, and justification sub-scores.

Input / output format

Input: Complex factual claims requiring multi-hop reasoning, accompanied by relevant documents/evidence from HOVER and FEVERIOUS.

Output: Model-generated responses containing claim extraction, evidence retrieval, and justification text. Evaluated via pairwise arena-style comparisons.

Scoring recipe

judgments = []
for claim in claims:
    responses = sample_8_responses(model, claim)
    for r1, r2 in pairwise(responses):
        verdict = llm_judge.compare(r1, r2)
        judgments.append(verdict)
accuracy = (sum(judgments) / len(judgments)) * 100
return average_over_3_runs(accuracy)

Common pitfalls

• Relies on LLM judges for pairwise comparisons rather than direct ground-truth labels, which may introduce systematic judge bias.
• Only 200 claims per dataset are sampled, potentially limiting coverage of the full HOVER/FEVERIOUS distribution.
• Fixed temperature of 0.0 ensures reproducibility but does not reflect typical inference conditions.

Evidence (verbatim from paper)

We randomly selected a total of 400 complex claims, each 200 from the datasets mentioned in §IV-A. For each claim we sampled 8 target model responses per task for pairwise comparisons to ensure diversity of samples in model comparisons as well as to maintain a controllable number of total battles, following the combinatorial coverage theory [50]. In the selected 400 claims, 85 are correctly predicted by all the selected target models, which are then semantically reversed and evolved. The final arena-styled judgment results in about 13,000 valid judgments, with each target LLM participating in approximately 1,600 comparisons, roughly 104 times for each model pair on average. Compared results are averaged over three random 3 runs. ACC. (%) DENOTES THE ACCURACY PERCENTAGE.

Citation

@misc{lin2026factarena,
  title={Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking},
  author={Lin et al. (2026)},
  year={2026},
  note={arXiv:2601.02669}
}

• arXiv: 2601.02669