Graphcheck Factcheck Eval

graphcheck-factcheck-eval

GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking — Yingjian Chen et al. (arXiv:2502.16514, 2025)

What this evaluates

This evaluation probes a model's ability to perform multihop fact-checking over long-form documents and open-domain QA contexts. It measures how well the system identifies factual inconsistencies or supports claims by reasoning over complex, lengthy grounding texts across general and medical domains.

Datasets

• AggreFact-CNN — total ?; splits: test (-1)
• AggreFact-Xsum — total ?; splits: test (-1)
• Summeval — total ?; splits: test (-1)
• ExpertQA — total ?; splits: test (-1)
• COVID-Fact — total ?; splits: test (-1)
• SCIFact — total ?; splits: test (-1)
• PubHealth — total ?; splits: test (-1)

Metrics

• balanced accuracy (primary) — range: percent
  • The average of recall scores computed per class: (1/C) * Σ (TP_c / (TP_c + FN_c)), where C is the number of classes. This metric is used to mitigate performance skew caused by class imbalance in the benchmarks.

Input / output format

Input: A claim and its corresponding grounding document (long-form text or QA context).

Output: A binary classification label indicating whether the claim is supported/refuted or true/false.

Scoring recipe

def balanced_accuracy(predictions, gold):
    classes = np.unique(gold)
    recalls = []
    for c in classes:
        mask = gold == c
        recalls.append(np.mean(predictions[mask] == c))
    return np.mean(recalls) * 100

Common pitfalls

• Standard accuracy is misleading due to class imbalance in several benchmarks; balanced accuracy must be used instead.
• OverallAvg. is computed as the mean of per-dataset balanced accuracies, which can obscure performance variations on specific domains like medical fact-checking.

Evidence (verbatim from paper)

Considering the data imbalance in some benchmarks, models biased towards a particular class in predictions may not reflect their true performance. To address this, we follow previous approachedLiu et al. ([2023b]); Tang et al. ([2023]) and calculate balanced accuracy (BAcc).

Citation

@misc{chen2025graphcheck,
  title={GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking},
  author={Yingjian Chen et al.},
  year={2025},
  note={arXiv:2502.16514}
}

• arXiv: 2502.16514