Pubhealth Fact Checking Eval

pubhealth-fact-checking-eval

Explainable Automated Fact-Checking for Public Health Claims — Kotonya et al. (2020) (arXiv:2010.09926, 2020)

What this evaluates

This benchmark evaluates automated fact-checking systems on public health claims, measuring both their ability to predict the veracity of a claim and the quality of the generated explanations. It probes domain-specific reasoning, extractive-abstractive explanation generation, and formal coherence properties like consistency and relevance.

Datasets

• PUBHEALTH — total 11832; splits: train (9466), val (1183), test (1183); repo https://github.com/neemakot/Health-Fact-Checking

Metrics

• macroF1 (primary) — range: percent
  • Macro-averaged F1 score across all classes (TRUE/FALSE).
• ROUGE-1 F1 — range: percent
  • F1 score for unigram overlap between generated and gold explanations.
• ROUGE-2 F1 — range: percent
  • F1 score for bigram overlap between generated and gold explanations.
• ROUGE-L F1 — range: percent
  • F1 score for longest common subsequence overlap between generated and gold explanations.
• Strong Global Coherence — range: percent
  • Percentage of explanations where every sentence entails the claim.
• Weak Global Coherence — range: percent
  • Percentage of explanations where no sentence contradicts the claim.
• Local Coherence — range: percent
  • Percentage of explanations where no two sentences contradict each other.

Input / output format

Input: A public health claim text and its associated evidence or article text.

Output: A veracity label (TRUE or FALSE) and a generated explanation text.

Scoring recipe

def score(predictions, golds, gold_explanations):
    preds = [1 if p == 'TRUE' else 0 for p in predictions]
    golds_bin = [1 if g == 'TRUE' else 0 for g in golds]
    macro_f1 = f1_score(golds_bin, preds, average='macro') * 100
    r1 = rouge_f1(predictions, gold_explanations, n=1)
    r2 = rouge_f1(predictions, gold_explanations, n=2)
    rl = rouge_f1(predictions, gold_explanations, n='L')
    def eval_sgc(exp, claim): return all(entails(e, claim) for e in exp)
    def eval_wgc(exp, claim): return all(not entails(e, neg(claim)) for e in exp)
    def eval_lc(exp):
        sents = exp.split('. ')
        return all(not entails(a, neg(b)) for a in sents for b in sents)
    return macro_f1, r1, r2, rl, eval_sgc, eval_wgc, eval_lc

Common pitfalls

• ROUGE scores are a poor proxy for explanation quality because generated explanations are heterogeneous and not directly comparable to gold standards.
• Computational NLI evaluation reliably approximates weak global and local coherence but fails to approximate strong global coherence, as entailment is a weak proxy for the required property.
• Human evaluation was conducted on a small sample (25 test examples with 5 annotators), limiting statistical reliability.

Evidence (verbatim from paper)

We evaluated veracity prediction using macroF1, precision, recall and accuracy metrics as shown in Table 4.

Citation

@misc{kotonya2020explainable,
  title={Explainable Automated Fact-Checking for Public Health Claims},
  author={Kotonya et al. (2020)},
  year={2020},
  note={arXiv:2010.09926}
}

• arXiv: 2010.09926