LLM Qe Fact Verification Eval

llm-qe-fact-verification-eval

Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion — Yejun Yoon et al. (2025) (arXiv:2504.14175, 2025)

What this evaluates

Evaluates LLM-based query expansion methods for evidence retrieval and claim verification. It measures retrieval quality and final verdict accuracy, while also analyzing whether generated documents contain sentences entailed by ground-truth evidence to detect knowledge leakage.

Datasets

• FEVER — total ?; splits: test (-1)
• SciFact — total ?; splits: test (-1)
• AVeriTeC — total ?; splits: test (-1)

Metrics

• Recall@5 (primary) — range: [0, 1]
  • Fraction of gold evidence sentences correctly retrieved in the top-5 results.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5, measuring ranked retrieval quality.
• F1 (primary) — range: [0, 1]
  • Macro F1 score for the final verdict prediction (True/False/NotEnoughInfo) using retrieved evidence.
• METEOR — range: [0, 100]
  • Token-level similarity between retrieved and reference evidence sentences, used for AVeriTeC.
• BERTScore — range: [0, 1]
  • Embedding-level similarity between retrieved and reference evidence sentences, used for AVeriTeC.

Input / output format

Input: Claim query (c) and a knowledge store (K) containing candidate evidence sentences.

Output: Top-5 retrieved evidence sentences; a predicted verdict (True, False, or NotEnoughInfo).

Scoring recipe

def evaluate(query, retrieved_top5, gold_evidence, gold_verdict):
    recall = len(set(retrieved_top5) & set(gold_evidence)) / len(gold_evidence)
    ndcg = compute_ndcg(retrieved_top5, gold_evidence)
    f1 = macro_f1(predict_verdict(query, retrieved_top5), gold_verdict)
    meteor = compute_meteor(retrieved_top5, gold_evidence)
    bertscore = compute_bertscore(retrieved_top5, gold_evidence)
    return recall, ndcg, f1, meteor, bertscore

Common pitfalls

• AVeriTeC uses human-written evidence rather than extracts from a shared knowledge store, requiring sentence-level similarity metrics (METEOR, BERTScore) instead of exact-match retrieval metrics.
• Performance gains from LLM-based query expansion are heavily driven by knowledge leakage; scores on claims where generated documents contain entailed sentences are significantly inflated compared to unmatched claims.
• Retrieval metrics are computed on the top-5 retrieved sentences, but the final verdict prediction uses all 5 retrieved pieces of evidence as context for a separate LLM.

Evidence (verbatim from paper)

For evidence retrieval, we used Recall@k and NDCG@k (k=5) as evaluation metrics on the FEVER and SciFact datasets, where both the ground-truth evidence E and retrieved evidence ě come from the knowledge store K. In contrast, ě in AveriTeC consists of human-written evidence rather than extracts from K. Therefore, following previous studies, we applied the Hungarian algorithm with METEOR and BERTScore on the top five retrieved sentences, computing token-level and embedding-level similarity, respectively, based on a binary assignment between generated and reference sequences. For verdict prediction, we used GPT-4o-mini with the five retrieved evidence and evaluated performance using macro F1.

Citation

@misc{yoon2025hypothetical,
  title={Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion},
  author={Yejun Yoon et al. (2025)},
  year={2025},
  note={arXiv:2504.14175}
}

• arXiv: 2504.14175