Factcheck Kg Validation Eval

factcheck-kg-validation-eval

Benchmarking Large Language Models for Knowledge Graph Validation — Shami et al. (2026) (arXiv:2602.10748, 2026)

What this evaluates

Evaluates large language models' ability to validate factual claims in knowledge graphs by classifying triples as true or false. It probes internal knowledge retrieval, retrieval-augmented generation (RAG) with external search results, and multi-model consensus strategies for fact-checking reliability.

Datasets

• FactBench — total 2800; splits: test (-1); HF FactCheck-AI/FactCheck; repo https://github.com/FactCheck-AI/FactCheck
• YAGO — total 1386; splits: test (-1); HF FactCheck-AI/FactCheck; repo https://github.com/FactCheck-AI/FactCheck
• DBpedia — total 9344; splits: test (-1); HF FactCheck-AI/FactCheck; repo https://github.com/FactCheck-AI/FactCheck

Metrics

• Class-wise F1 Score (primary) — range: [0, 1]
  • F1(c) = 2 * Precision(c) * Recall(c) / (Precision(c) + Recall(c)), calculated independently for True and False classes to handle class imbalance.
• Consensus Alignment — range: [0, 1]
  • CA_M = (1/|G|) * sum(I(response(M,t) == majorityVote(t))) over all facts G, measuring agreement between a model's predictions and the majority vote across all evaluated models.
• Average Response Time — range: other
  • IQR-filtered mean of per-fact response times in seconds, excluding outliers outside Q1-1.5IQR and Q3+1.5IQR.

Input / output format

Input: A knowledge graph triple (subject, predicate, object) or a generated question derived from it, optionally accompanied by a fixed set of retrieved web documents (for RAG evaluation).

Output: Binary classification label: 'True' or 'False' indicating whether the fact/triple is valid.

Scoring recipe

def compute_metrics(predictions, gold, ensemble_preds=None):
    f1_scores = {}
    for c in [1, 0]:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_scores[c] = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    ca = 0
    if ensemble_preds is not None:
        majority = [max(set(row), key=row.count) for row in zip(*ensemble_preds)]
        ca = sum(1 for p, m in zip(predictions, majority) if p == m) / len(predictions)
    return f1_scores, ca

Common pitfalls

• YAGO has a 99% gold accuracy rate, so models may simply predict 'True' for all instances, inflating accuracy while failing to detect false facts. Class-wise F1 is required to expose this bias.
• Consensus Alignment measures agreement with the majority vote across all evaluated models, not agreement with ground truth. Misinterpreting it as a correctness metric leads to flawed conclusions.
• Live web search results change over time, breaking reproducibility. The benchmark provides a mock API with pre-fetched SERP results; using live queries instead violates the evaluation protocol.

Evidence (verbatim from paper)

To assess the effectiveness of the considered fact validation strategies, we focus on two key measures: Class-wise F1 Score and Consensus Alignment. These measures are chosen to account for class imbalance, capture per-class performance, and evaluate agreement for multi-model consensus approaches. The F1 score for a given class c∈{T,F} is defined as: F1(c)=2⋅Precision(c)⋅Recall(c)/(Precision(c)+Recall(c), where Precision(c) and Recall(c) denote the precision and recall calculated specifically for class c.

Citation

@misc{shami2026benchmarking,
  title={Benchmarking Large Language Models for Knowledge Graph Validation},
  author={Shami et al. (2026)},
  year={2026},
  note={arXiv:2602.10748}
}

• arXiv: 2602.10748