Factlens Eval

factlens-eval

FactLens: Benchmarking Fine-Grained Fact Verification — Mitra et al. (2024) (arXiv:2411.05980, 2024)

What this evaluates

Evaluates the quality of fine-grained claim decomposition and the downstream impact of sub-claim quality on fact verification performance. It probes an LLM's ability to break complex claims into atomic, sufficient, and non-fabricated sub-claims, and measures how these sub-claim properties correlate with verification accuracy.

Datasets

• CoverBench — total 733; splits: eval (733); repo https://github.com/megagonlabs/factlens

Metrics

• atomicity — range: [1, 3]
  • Categorical score (non-atomic-2, non-atomic-1, atomic) mapped to numerical values (1, 2, 3). Average across instances.
• sufficiency — range: other
  • Categorical score (low, medium, high) averaged across instances.
• fabrication — range: other
  • Categorical score (low, medium, high) averaged across instances.
• coverage — range: other
  • Categorical score (low, medium, high) averaged across instances.
• redundancy — range: other
  • Categorical score (low, medium, high) averaged across instances.
• readability — range: other
  • Categorical score (low, medium, high) averaged across instances.
• verification F1 (primary) — range: [0, 1]
  • F1-score of the aggregated claim-level verification label against ground-truth claim labels.

Input / output format

Input: Original complex claim and associated evidence/context from CoverBench.

Output: List of decomposed sub-claims, per-sub-claim verification labels (true/false), and an aggregated claim-level verification label.

Scoring recipe

sub_claims = decompose_claim(claim)
sub_labels = [verify(sub_claim, evidence) for sub_claim in sub_claims]
claim_label = 'false' if any(l == 'false' for l in sub_labels) else 'true'

# FactLens Evaluator metrics (mapped to 1-3 or low/med/high)
atomicity_score = map_to_1_2_3(atomicity_label)
sufficiency_score = map_to_1_2_3(sufficiency_label)
# ... similarly for fabrication, coverage, redundancy, readability

# Aggregation
avg_atomicity = mean(atomicity_score across 733 instances)
verification_f1 = f1_score(gold_claim_labels, claim_label)

Common pitfalls

• Sub-claim verification labels are aggregated to the claim level using a strict 'any-false' rule, which may penalize models for minor sub-claim errors even if the overall claim is true.
• Metrics like atomicity and fabrication use categorical low/medium/high scores that are manually mapped to numerical values (1-3) for averaging, which can obscure distribution details.
• The evaluation isolates decomposition and verification from evidence retrieval, meaning performance does not reflect end-to-end retrieval-augmented pipelines.

Evidence (verbatim from paper)

For each instance, we map the ‘low’, ‘medium’, ‘high’ scores (‘non-atomic-2’, ‘non-atomic-1’, ‘atomic’ for atomicity) to numerical values (1, 2, 3 respectively), and report the average for each metric across all 733 instances in the CoverBench dataset. To compare the performance of fine-grained verification labels (for each sub-claim) with the holistic verification label, we aggregate the fine-grained labels as false if at least one of the fine-grained labels is also false; otherwise, the claim is considered true.

Citation

@misc{mitra2024factlens,
  title={FactLens: Benchmarking Fine-Grained Fact Verification},
  author={Mitra et al. (2024)},
  year={2024},
  note={arXiv:2411.05980}
}

• arXiv: 2411.05980