News Claim Verification Eval

news-claim-verification-eval

Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM — Zhang et al. (2024) (arXiv:2404.17283, 2024)

What this evaluates

Evaluates a model's ability to verify the veracity of real-world news claims by retrieving relevant evidence documents and predicting a multi-class label. It probes the system's capacity for evidence selection, claim decomposition, and fact-checking under black-box LLM constraints.

Datasets

• RAWFC — total 2012; splits: train (1612), valid (200), test (200)
• LIAR-RAW — total 12590; splits: train (10065), valid (1274), test (1251)

Metrics

• macro-average F1 (primary) — range: percent
  • Macro-averaged precision, recall, and F1 score across all veracity classes. F1 is computed as F1 = 2 * (Precision * Recall) / (Precision + Recall). Macro-averaging treats all classes equally regardless of frequency.

Input / output format

Input: News claim text, optionally decomposed into intermediate questions, and a set of retrieved evidence documents (top-K or top-20 from a dense retrieval index).

Output: A veracity class label prediction from the dataset's predefined set (e.g., True, False, Half for RAWFC; True, Mostly-true, Half-true, Barely-true, False, Pants-fire for LIAR-RAW).

Scoring recipe

def compute_macro_f1(y_true, y_pred, classes):
    precisions, recalls, f1s = [], [], []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
        precisions.append(p)
        recalls.append(r)
        f1s.append(f1)
    return sum(precisions) / len(classes), sum(recalls) / len(classes), sum(f1s) / len(classes)

Common pitfalls

• Ground-truth leakage in original datasets can artificially inflate performance; the authors explicitly remove leaked documents before evaluation.
• Using Wikipedia-based datasets (e.g., FEVER) risks data contamination since Wikipedia is heavily present in LLM pretraining corpora.
• Macro-averaging is required due to class imbalance across the six veracity categories in LIAR-RAW.

Evidence (verbatim from paper)

Following Yang et al. ([2022b]), we use macro-average precision ($P$), recall ($R$), and $F_{1}$ ($F_{1}=\frac{2RP}{R+P}$) scores for evaluation. We utilize the supplied division of train-validate-test for both data sets. Specifically, they employed an 8/1/1 split ratio for the train/validation/test for the two datasets. Therefore, the corresponding number of samples in the RAWFC data for train/valid/test is 1,612/200/200, and the number of samples in the LIAR-RAW dataset is 10,065/1,274/1,251.

Citation

@misc{zhang2024reinforcementretrieval,
  title={Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLM},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2404.17283}
}

• arXiv: 2404.17283