Cfever Eval

cfever-eval

CFEVER: A Chinese Fact Extraction and VERification Dataset — Lin et al. (2024) (arXiv:2402.13025, 2024)

What this evaluates

Evaluates a model's ability to retrieve supporting or refuting evidence from Chinese Wikipedia documents and sentences, and subsequently verify claims by predicting their factual status (Supports, Refutes, or Not Enough Info) based on the retrieved evidence.

Datasets

• CFEVER — total 30012; splits: test (-1)

Metrics

• Recall — range: percent
  • Percentage of instances where the model correctly predicts at least one complete set of evidence pages or sentences.
• Accuracy — range: percent
  • Percentage of claims correctly classified as Supports, Refutes, or Not Enough Info.
• FEVER Score (primary) — range: percent
  • Strict measure of accuracy requiring both the correct claim label and at least one complete set of correctly retrieved evidence sentences for each claim.

Input / output format

Input: A factual claim paired with candidate evidence documents and sentences from Chinese Wikipedia.

Output: A predicted label (Supports, Refutes, or Not Enough Info) and a set of retrieved evidence sentences/pages.

Scoring recipe

fever_correct = 0
recall_correct = 0
for instance in dataset:
    pred_evidence = model.predict_evidence(instance.claim)
    gold_evidence = instance.gold_evidence
    pred_label = model.predict_label(instance.claim, pred_evidence)
    gold_label = instance.gold_label
    evidence_match = any(set(pred) == set(gold) for pred in pred_evidence for gold in gold_evidence)
    if evidence_match:
        recall_correct += 1
    if evidence_match and pred_label == gold_label:
        fever_correct += 1
recall = (recall_correct / len(dataset)) * 100
fever_score = (fever_correct / len(dataset)) * 100

Common pitfalls

• FEVER Score is stricter than standard accuracy; it requires both the correct classification label AND at least one complete set of correctly retrieved evidence, not just label accuracy.
• Recall is defined as predicting 'at least one complete set' of evidence, meaning partial matches or single-sentence matches do not count toward the score.
• Evaluation relies on official scoring tools from Thorne et al. (2018b) and DeHaven & Scott (2023); using custom or approximate matching scripts will yield non-comparable results.

Evidence (verbatim from paper)

For document retrieval and sentence retrieval, we report the performance in recall (%). Our recall evaluation metric is designed to assess the model’s ability to correctly predict at least one complete set of evidence pages during document retrieval and, similarly, at least one complete set of evidence sentences during sentence retrieval, for each data instance. For claim verification in RTE, following Thorne et al. (2018b), we report performance in accuracy (%) and FEVER Score (%). The latter is a strict measure of accuracy, requiring a model to correctly predict at least one complete evidence set for each claim.

Citation

@misc{lin2024cfever,
  title={CFEVER: A Chinese Fact Extraction and VERification Dataset},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2402.13025}
}

• arXiv: 2402.13025