Faviq Eval

faviq-eval

FaVIQ: FAct Verification from Information-seeking Questions — Park et al. (2021) (arXiv:2107.02153, 2021)

What this evaluates

Evaluates a model's ability to verify factual claims against retrieved evidence, specifically focusing on claims derived from ambiguous information-seeking questions. It also measures the effectiveness of transfer learning from crowdsourced fact-checking data to professional fact-checking benchmarks.

Datasets

• FaVIQ — total 188000; splits: train (-1), dev (-1), test (-1); repo https://github.com/faviq/faviq
• Snopes — total 6422; splits: test (-1), dev (-1)
• SciFACT — total 1109; splits: test (-1), dev (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly predicted verdicts (support or refute) out of the total number of claims. The task is framed as a 2-way classification where 'not enough info' (NEI) is merged into 'refute'.

Input / output format

Input: A claim concatenated with up to k retrieved passages (e.g., k=3 Wikipedia passages or k=10 scientific abstracts).

Output: Generate either 'support' or 'refute'.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Models trained on FEVER perform poorly on FaVIQ in zero-shot (near random), highlighting a severe domain shift and lack of generalization to ambiguous claims.
• Retrieval errors account for 38% of failures, often because claims have low lexical overlap with the correct evidence passages.
• The original 3-way classification (Support/Refute/NEI) is converted to 2-way by merging NEI into Refute, which changes the evaluation landscape compared to prior work.

Evidence (verbatim from paper)

The overall accuracy of the baselines is low, despite their high performance on FEVER. The zero-shot performance is barely better than random guessing, indicating that the model trained on FEVER is not able to generalize to our more challenging data.

Citation

@misc{park2021faviq,
  title={FaVIQ: FAct Verification from Information-seeking Questions},
  author={Park et al. (2021)},
  year={2021},
  note={arXiv:2107.02153}
}

• arXiv: 2107.02153