Trendfact Eval

trendfact-eval

TrendFact: A Benchmark for Explainable Hotspot Perception in Fact-Checking with Natural Language Explanation — Zhang et al. (2024) (arXiv:2410.15135, 2024)

What this evaluates

Evaluates fact-checking systems across three sub-tasks: retrieving relevant evidence, verifying claim truthfulness, and generating natural language explanations. It specifically probes a model's Hotspot Perception Ability (HPA) by measuring how effectively it allocates reasoning effort and computational resources based on the real-world influence of trending claims.

Datasets

• TrendFact — total 7643; splits: test (-1); repo https://github.com/zxc123cc/TrendFact

Metrics

• R@k (k=1,2,3,5) — range: [0, 1]
  • Recall at k: fraction of gold evidence items successfully retrieved within the top-k ranked results from the evidence library.
• F1-macro (primary) — range: [0, 1]
  • Macro-averaged F1 score across verification classes. Calculated as the unweighted mean of per-class F1 scores (2 * precision * recall / (precision + recall)).
• HCPI — range: [0, 1]
  • Hotspot Claim Perception Index: evaluates how well a system allocates reasoning effort based on claim influence, factoring in verification accuracy and reasoning efficiency/time.
• ECS — range: [0, 1]
  • Explanation Consistency Score: measures the alignment between the generated explanation and the model's internal reasoning process.
• BLEU-4 — range: [0, 1]
  • Standard 4-gram BLEU score comparing generated explanations to reference explanations.
• ROUGE-1/2/L — range: [0, 1]
  • ROUGE scores measuring overlap of unigrams, bigrams, and longest common subsequence between generated and reference explanations.
• BERTScore — range: [0, 1]
  • F1 score computed using contextual embeddings from BERT to measure semantic similarity between generated and reference explanations.

Input / output format

Input: Claim text, associated metadata (e.g., influence/hotspot score), and a dynamic evidence library for retrieval.

Output: Top-k retrieved evidence snippets, a verification label (e.g., True/False/Unsupported), and a natural language explanation justifying the verdict.

Scoring recipe

def score(predictions, gold):
    # Retrieval
    r_at_k = [sum(1 for p in preds[:k] if p in gold_evidence) / len(gold_evidence) for k in [1,2,3,5]]
    # Verification
    tp = sum(1 for p, g in zip(preds, gold) if p == g == 'True')
    fp = sum(1 for p, g in zip(preds, gold) if p == 'True' and g != 'True')
    fn = sum(1 for p, g in zip(preds, gold) if p != 'True' and g == 'True')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_macro = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    # Explanation & HPA
    ecs = compute_consistency_score(predictions.explanations, predictions.reasoning_trace)
    hcp = compute_hotspot_perception(predictions.f1, predictions.reasoning_effort, gold.influence_scores)
    return {'R@k': r_at_k, 'F1-macro': f1_macro, 'ECS': ecs, 'HCPI': hcp}

Common pitfalls

• HCPI is derived from verification results and reasoning efficiency, so errors in the verification step directly cascade and penalize the hotspot perception score.
• ECS evaluates consistency with internal reasoning rather than factual correctness, which may artificially inflate scores for models that generate verbose but logically aligned explanations.
• The evidence library is dynamically augmented and contains challenging, similar-looking items, making R@k scores incomparable to standard retrieval benchmarks.

Evidence (verbatim from paper)

For evidence retrieval task, we choose R@k, where k=1,2,3,5. For verification task, we choose F1-macro, Precision, Recall, and Accuracy. For explanation generation task, in addition to ECS, we also employ BLEU-4, ROUGE-(1, 2, L), and BERTScore. For assessing the HPA of fact-checking systems, we employ HCPI.

Citation

@misc{zhang2024trendfact,
  title={TrendFact: A Benchmark for Explainable Hotspot Perception in Fact-Checking with Natural Language Explanation},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2410.15135}
}

• arXiv: 2410.15135