Livefact Eval

livefact-eval

LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection — Cheng Xu et al. (2026) (arXiv:2604.04815, 2026)

What this evaluates

Evaluates LLMs on fake news detection under dynamic, time-evolving evidence streams. It probes both binary classification capability (Real/Fake) and reasoning capability (handling ambiguity when evidence is incomplete), while explicitly measuring benchmark data contamination and epistemic humility.

Datasets

• LiveFact November 2025 dataset — total ?; splits: test (-1); repo https://github.com/bebxy/livefact

Metrics

• average_score (primary) — range: percent
  • Average of Classification Accuracy and Inference Accuracy across the dataset.
• reasoning_gap — range: percent
  • Inference Accuracy minus Classification Accuracy at temporal offset δ=-3.

Input / output format

Input: A news claim paired with a dynamic, time-stamped evidence stream, evaluated at specific temporal offsets (δ) relative to the claim's publication.

Output: A single label enclosed in double brackets, e.g., [[Real]], [[Fake]], or [[Ambiguous]].

Scoring recipe

def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return (correct / len(gold)) * 100

class_acc = compute_accuracy(class_preds, class_gold)
inf_acc = compute_accuracy(inf_preds, inf_gold)
average_score = (class_acc + inf_acc) / 2
reasoning_gap = inf_acc - class_acc

Common pitfalls

• Base models fail due to format non-compliance (not outputting [[LABEL]]), resulting in near-random parsed predictions rather than true reasoning failures.
• Reasoning-tuned models require extended output limits (1024 tokens) to generate Chain-of-Thought; truncating at 128 tokens yields near-zero accuracy despite strong underlying capabilities.
• Classification mode forces definitive verdicts even when evidence is absent, artificially penalizing models that should express uncertainty.

Evidence (verbatim from paper)

The Qwen3-235B-A22B-Instruct model not only leads the open-source sector but also outperforms proprietary flagship models, including gpt-5.1, achieving the highest average score of 72.40%.

Citation

@misc{xu2026livefact,
  title={LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection},
  author={Cheng Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.04815}
}

• arXiv: 2604.04815