User Claim Distribution Eval

user-claim-distribution-eval

What do people want to fact-check? — Ghafouri et al. (2026) (arXiv:2602.10935, 2026)

What this evaluates

This evaluation probes the semantic, epistemic, and verifiability characteristics of real-world user-submitted fact-checking requests. It measures how public demand for verification aligns with or diverges from synthetic benchmark distributions, highlighting gaps in current misinformation evaluation corpora.

Datasets

• User Fact-Checking Claims Dataset — total ?; splits: full (-1)

Metrics

• Shannon Entropy — range: other
  • H_i^(d) = -sum_{k=1}^{K_d} p_{ik}^{(d)} log p_{ik}^{(d)}, where p_{ik}^{(d)} is the proportion of user i's claims in category k of dimension d. Measures user-level diversity across classification dimensions.
• Veracity Score (primary) — range: [0, 1]
  • A continuous measure from 0 (false) to 1 (true) reflecting how definitively a claim can be resolved. Computed per claim and averaged across dimensions.
• Classification Proportion — range: percent
  • Percentage of claims falling into each of the five semantic dimensions: Domain, Epistemic Type, Verifiability Level, Target Entity, and Temporal Orientation.

Input / output format

Input: Raw text of user-submitted fact-checking claims.

Output: Five categorical labels (Domain, Epistemic Type, Verifiability, Target, Temporal), a continuous veracity score (0-1), and per-user entropy scores across dimensions.

Scoring recipe

def compute_metrics(claims, classifiers):
    # 1. Classify each claim across 5 dimensions
    labels = {dim: classifier(cl) for dim, classifier in classifiers.items()}
    # 2. Compute continuous veracity score (0-1) per claim
    veracity = veracity_model(cl)
    # 3. Compute user-level Shannon entropy per dimension
    user_dist = count_proportions(user_claims, dimension)
    entropy = -sum(p * log(p) for p in user_dist.values())
    return labels, veracity, entropy

Common pitfalls

• Assuming veracity is binary; the paper uses a continuous 0-1 scale reflecting resolution definitiveness.
• Overlooking that ~25% of claims are normative or unverifiable, which standard benchmarks like FEVER typically exclude or misrepresent.
• Treating the dataset as a model training benchmark rather than a demand-side distribution analysis.

Evidence (verbatim from paper)

We then compared the average veracity score in each dataset, a continuous measure from 0 (false) to 1 (true) reflecting how definitively a claim can be resolved, and find that user-submitted claims cluster much closer to the ambiguous middle (mean = 0.58, SD = 0.38) than FEVER’s sharply polarized claims (mean = 0.38, SD = 0.47), showing that real users most often seek verification for grey-area statements rather than the clearly true or false claims that dominate benchmark datasets.

Citation

@misc{ghafouri2026what,
  title={What do people want to fact-check?},
  author={Ghafouri et al. (2026)},
  year={2026},
  note={arXiv:2602.10935}
}

• arXiv: 2602.10935