Hover Eval

hover-eval

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification — Jiang et al. (2020) (arXiv:2011.03088, 2020)

What this evaluates

Probes a model's ability to perform many-hop fact verification by retrieving supporting evidence from multiple Wikipedia articles and determining whether a given claim is supported or not supported. It specifically tests long-range dependency reasoning, coreference resolution, and the ability to avoid semantic matching shortcuts that degrade as hop count increases.

Datasets

• HOVER — total ?; splits: dev (-1), test (-1)

Metrics

• claim verification accuracy (primary) — range: [0, 1]
  • Percentage of claims correctly classified as SUPPORTED or NOT-SUPPORTED.
• document retrieval exact-match — range: [0, 1]
  • Exact-match score comparing predicted document-level evidence to ground-truth evidence documents.
• document retrieval F1 scores — range: [0, 1]
  • F1 score comparing predicted document-level evidence to ground-truth evidence documents.
• sentence retrieval exact-match — range: [0, 1]
  • Exact-match score comparing predicted sentence-level evidence to ground-truth evidence sentences.
• sentence retrieval F1 scores — range: [0, 1]
  • F1 score comparing predicted sentence-level evidence to ground-truth evidence sentences.
• HOVER Score — range: percent
  • Percentage of examples where the model retrieves at least one supporting fact from every supporting document and predicts the correct claim label.

Input / output format

Input: A natural language claim and a set of candidate Wikipedia documents (and optionally sentences within those documents) from which evidence must be retrieved.

Output: Binary classification label: SUPPORTED or NOT-SUPPORTED. For retrieval subtasks, predicted document/sentence indices or text spans.

Scoring recipe

# Claim Verification Accuracy
correct = sum(1 for p, g in zip(pred_labels, gold_labels) if p == g)
accuracy = correct / len(gold_labels)

# HOVER Score
hover_correct = 0
for pred_docs, pred_sents, pred_label, gold_docs, gold_label in zip(pred_evidence_docs, pred_evidence_sents, pred_labels, gold_evidence_docs, gold_labels):
    # Must retrieve at least one supporting fact from every supporting document
    docs_covered = all(any(gd in pred_docs for gd in gold_docs))
    if docs_covered and (pred_label == gold_label):
        hover_correct += 1
hover_score = hover_correct / len(gold_labels)

Common pitfalls

• Models can achieve high accuracy by exploiting claim-only biases without retrieving evidence, masking poor retrieval performance.
• Performance degrades sharply as hop count increases (2 to 4), revealing that models often rely on single-hop word-matching shortcuts rather than true multi-hop reasoning.
• Dataset construction explicitly minimizes semantic overlap between claims and distractor documents to prevent trivial matching solutions.

Evidence (verbatim from paper)

We evaluate the final accuracy of the claim verification task to predict a claim as SUPPORTED or NOT-SUPPORTED. The document and sentence retrieval are evaluated by the exact-match and F1 scores between the predicted document/sentence-level evidence and the ground-truth evidence for the claim. We further propose the HOVER Score, which is the percentage of the examples where the model must retrieve at least one supporting fact from every supporting document and predict the correct label.

Citation

@misc{jiang2020hover,
  title={HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification},
  author={Jiang et al. (2020)},
  year={2020},
  note={arXiv:2011.03088}
}

• arXiv: 2011.03088