Poly Fever Eval

poly-fever-eval

Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models — Zhang et al. (2025) (arXiv:2503.16541, 2025)

What this evaluates

Evaluates large language models' ability to detect hallucinations by verifying factual claims across 11 languages. It probes cross-linguistic consistency, topic-aware fact-checking, and resistance to web-resource bias in multilingual settings.

Datasets

• Poly-FEVER — total 77973; splits: full (77973)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy computed over binary veracity labels (true/false). Calculated as the proportion of claims where the model's predicted label matches the ground truth label.

Input / output format

Input: A single textual claim provided in one of 11 languages (English, Mandarin Chinese, Hindi, Arabic, Bengali, Japanese, Korean, Tamil, Thai, Georgian, or Amharic).

Output: A binary veracity label: 'true' or 'false'.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(gold_labels)

Common pitfalls

• Original FEVER includes 'NotEnoughInfo' claims, but Poly-FEVER explicitly excludes them; evaluators must filter these out to avoid invalid comparisons.
• Non-English claims are machine-translated via Google Translate; poor translation quality in low-resource languages may be misattributed to model hallucination rather than linguistic artifacts.
• Topic distribution vectors are provided as metadata but are not part of the verification prompt; models may inadvertently learn topic priors that skew verification performance.

Evidence (verbatim from paper)

Poly-FEVER excludes claims labeled as NotEnoughInfo to focus on cases where factual accuracy can be measured directly. Each entry in Poly-FEVER contains four primary fields: ... Label: An annotated veracity label (true or false) indicating whether the claim aligns with established factual evidence.

Citation

@misc{zhang2025polyfever,
  title={Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.16541}
}

• arXiv: 2503.16541