# Cfever Eval

> Evaluates a model's ability to retrieve supporting or refuting evidence from Chinese Wikipedia documents and sentences, and subsequently verify claims by predicting their factual status (Supports, Refutes, or Not Enough Info) based on the retrieved evidence. Use when the user wants to benchmark on CFEVER, or asks about evaluating this task. Reports FEVER Score.

- Skill: `qhjqhj00/cfever-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/cfever-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/cfever-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/cfever-eval

---


# cfever-eval

> CFEVER: A Chinese Fact Extraction and VERification Dataset — Lin et al. (2024) (arXiv:2402.13025, 2024)

## What this evaluates

Evaluates a model's ability to retrieve supporting or refuting evidence from Chinese Wikipedia documents and sentences, and subsequently verify claims by predicting their factual status (Supports, Refutes, or Not Enough Info) based on the retrieved evidence.

## Datasets

- **CFEVER** — total 30012; splits: test (-1)

## Metrics

- `Recall` — range: percent
  - Percentage of instances where the model correctly predicts at least one complete set of evidence pages or sentences.
- `Accuracy` — range: percent
  - Percentage of claims correctly classified as Supports, Refutes, or Not Enough Info.
- `FEVER Score` **(primary)** — range: percent
  - Strict measure of accuracy requiring both the correct claim label and at least one complete set of correctly retrieved evidence sentences for each claim.

## Input / output format

**Input**: A factual claim paired with candidate evidence documents and sentences from Chinese Wikipedia.

**Output**: A predicted label (Supports, Refutes, or Not Enough Info) and a set of retrieved evidence sentences/pages.

## Scoring recipe

```python
fever_correct = 0
recall_correct = 0
for instance in dataset:
    pred_evidence = model.predict_evidence(instance.claim)
    gold_evidence = instance.gold_evidence
    pred_label = model.predict_label(instance.claim, pred_evidence)
    gold_label = instance.gold_label
    evidence_match = any(set(pred) == set(gold) for pred in pred_evidence for gold in gold_evidence)
    if evidence_match:
        recall_correct += 1
    if evidence_match and pred_label == gold_label:
        fever_correct += 1
recall = (recall_correct / len(dataset)) * 100
fever_score = (fever_correct / len(dataset)) * 100
```

## Common pitfalls

- FEVER Score is stricter than standard accuracy; it requires both the correct classification label AND at least one complete set of correctly retrieved evidence, not just label accuracy.
- Recall is defined as predicting 'at least one complete set' of evidence, meaning partial matches or single-sentence matches do not count toward the score.
- Evaluation relies on official scoring tools from Thorne et al. (2018b) and DeHaven & Scott (2023); using custom or approximate matching scripts will yield non-comparable results.

## Evidence (verbatim from paper)

> For document retrieval and sentence retrieval, we report the performance in recall (%). Our recall evaluation metric is designed to assess the model’s ability to correctly predict at least one complete set of evidence pages during document retrieval and, similarly, at least one complete set of evidence sentences during sentence retrieval, for each data instance. For claim verification in RTE, following Thorne et al. (2018b), we report performance in accuracy (%) and FEVER Score (%). The latter is a strict measure of accuracy, requiring a model to correctly predict at least one complete evidence set for each claim.

## Citation

```bibtex
@misc{lin2024cfever,
  title={CFEVER: A Chinese Fact Extraction and VERification Dataset},
  author={Lin et al. (2024)},
  year={2024},
  note={arXiv:2402.13025}
}
```

- arXiv: 2402.13025