Defame Fact Checking Eval

defame-fact-checking-eval

DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts — Braun et al. (2024) (arXiv:2412.10510, 2024)

What this evaluates

This evaluation probes a model's ability to dynamically retrieve and reason over multimodal evidence to verify open-domain claims. It tests zero-shot multimodal fact-checking across text-only, text-image, and out-of-context scenarios, measuring both classification accuracy and the quality of generated justifications.

Datasets

• AVeriTeC — total 500; splits: dev (500)
• MOCHEG — total 1689; splits: test (1689)
• VERITE — total 1001; splits: test (1001)
• ClaimReview2024+ — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted verdicts out of the total number of claims. For MOCHEG, this is equivalent to micro-F1 across the three classes (Refuted, NEI, Supported).
• Coherence — range: [1, 5]
  • Human-rated score on a Likert scale from 1 to 5 assessing whether the fact-check maintains a logical flow without contradictions or gaps.
• Completeness — range: [1, 5]
  • Human-rated score on a Likert scale from 1 to 5 assessing whether the verdict is sufficiently justified by the included evidence.

Input / output format

Input: Interleaved text and image claims. The model receives the claim text, associated images, and dynamically retrieved multimodal evidence (text and images) from simulated web search, reverse image search, geolocation, and image search tools.

Output: A structured verdict (e.g., Refuted, Supported, NEI, True, OOC, Miscaptioned, or Misleading) accompanied by a natural language justification/report detailing the evidence used.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

def compute_human_score(ratings):
    return sum(ratings) / len(ratings)

Common pitfalls

• Confusing semantically similar labels like 'Refuted', 'Misleading', and 'NEI', which leads to high misclassification rates even when evidence is correctly retrieved.
• Relying on parametric knowledge instead of dynamic retrieval causes severe performance drops on post-cutoff claims, as models hallucinate or default to 'NEI'.
• Failing to distinguish between 'Out-Of-Context' (OOC) and 'Miscaptioned' image scenarios, which require different levels of image-text alignment verification.

Evidence (verbatim from paper)

We evaluate our performance using accuracy—equivalent to micro-F1 (Appendix[J]). Following Papadopoulos et al. ([2024a]), we report accuracy for “True vs. OOC” and “True vs. Miscaptioned,” as well as a merged “True vs. False” setup.

Citation

@misc{braun2024defame,
  title={DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts},
  author={Braun et al. (2024)},
  year={2024},
  note={arXiv:2412.10510}
}

• arXiv: 2412.10510