Verifact Eval

verifact-eval

VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts — First Author et al. (2025) (arXiv:2505.09701, 2025)

What this evaluates

Evaluates the factual correctness of long-form LLM-generated responses by decomposing them into atomic facts, detecting and refining incomplete or missing information, and verifying each fact against external web evidence.

Datasets

• Long-form LLM responses — total ?; splits: test (-1)

Metrics

• Supported/Contradicted/Undecided classification accuracy (primary) — range: [0, 1]
  • Proportion of facts correctly classified into one of three categories (Supported, Contradicted, Undecided) relative to human-annotated ground truth. Calculated as correct predictions divided by total facts evaluated.

Input / output format

Input: A long-form LLM response, which is algorithmically decomposed into atomic facts. For verification, each fact is paired with retrieved web search snippets serving as evidence.

Output: Per fact: a categorical label from the set {Supported, Contradicted, Undecided}.

Scoring recipe

def score(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(predictions)

Common pitfalls

• The pipeline intentionally prioritizes recall over precision during the incomplete/missing fact detection step, retaining false positives to avoid missing errors.
• Verification relies on Google Search API snippets, which may be noisy, incomplete, or fail to cover niche claims.
• Decontextualization and refinement steps may inadvertently alter the original semantic meaning if the LLM fails to preserve contextual dependencies.

Evidence (verbatim from paper)

We then adopt the design improvements of VERIFY Bayat et al. ([2025]), using Llama 3.3-70B to classify the fact as Supported, Contradicted, or Undecided based on the gathered evidence.

Citation

@misc{firstauthor2025verifact,
  title={VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts},
  author={First Author et al. (2025)},
  year={2025},
  note={arXiv:2505.09701}
}

• arXiv: 2505.09701