qhjqhj00 fgveribench-eval
DiVA: Fine-grained Factuality Verification with Agentic-Discriminative Verifier — Hui Huang, Muyun Yang, Yuki Arase (2026) (arXiv:2601.03605, 2026)
What this evaluates
Evaluates an LLM verifier's ability to rank multiple candidate answers by their factual correctness and error severity across single-hop and multi-hop questions, using external knowledge retrieval and fine-grained scoring.
Datasets
- FGVeriBench — total ?; splits: test (-1); repo https://github.com/HuihuiChyan/FactVeri-SFT
Metrics
Kendall-tau(primary) — range: [-1, 1]- Kendall rank correlation coefficient measuring the ordinal association between predicted factuality scores and ground-truth scores. Ranges from -1 to 1, with 1 indicating perfect agreement.
P@1— range: [0, 1]- Precision at rank 1; measures whether the verifier correctly identifies the most factual answer as the top-ranked candidate.
Input / output format
Input: A question, a set of candidate answers, and retrieved external evidence (from WebSearch and/or LocalSearch).
Output: A fine-grained factuality score for each candidate answer, or a binary correct/incorrect label for binary verification settings.
Scoring recipe
def compute_ktau(pred_scores, gold_scores):
return kendalltau(pred_scores, gold_scores).correlation
def compute_p1(pred_scores, gold_scores):
return 1.0 if np.argmax(pred_scores) == np.argmax(gold_scores) else 0.0
Common pitfalls
- Relying solely on the LLM's internal parametric knowledge without external retrieval causes significant performance drops, as static weights cannot update over time.
- Fine-grained verification requires ranking candidates by error severity rather than simple binary correctness; models optimized only for binary accuracy often fail to capture nuanced factuality differences.
Evidence (verbatim from paper)
Table 7: Kendall-tau correlation coefficient of different knowledge sources on FGVeriBench.
Citation
@misc{huang2026diva,
title={DiVA: Fine-grained Factuality Verification with Agentic-Discriminative Verifier},
author={Hui Huang, Muyun Yang, Yuki Arase (2026)},
year={2026},
note={arXiv:2601.03605}
}
- arXiv: 2601.03605