Hotpotqa Eval

hotpotqa-eval

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering — Zhilin Yang et al. (2018) (arXiv:1809.09600, 2018)

What this evaluates

This benchmark evaluates a model's ability to perform multi-hop question answering by reasoning across multiple documents. It specifically probes explainability through supporting fact prediction and tests robustness against distractor paragraphs and large-scale retrieval contexts.

Datasets

• HotpotQA — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score between predicted and gold answer spans. Computed as 2 * (precision * recall) / (precision + recall).
• exact match (EM) — range: [0, 1]
  • Binary score: 1 if the predicted answer exactly matches the gold answer string, 0 otherwise.
• Joint F1 — range: [0, 1]
  • Combines answer and supporting fact performance: P^(joint) = P^(ans) * P^(sup), R^(joint) = R^(ans) * R^(sup), Joint F1 = 2 * P^(joint) * R^(joint) / (P^(joint) + R^(joint)).
• Joint EM — range: [0, 1]
  • Binary score: 1 only if both the answer span and the set of supporting facts achieve exact match, otherwise 0.

Input / output format

Input: A question paired with a set of context paragraphs. In the distractor setting, exactly 10 paragraphs (1 containing the answer, 9 distractors). In the full wiki setting, the top 10 paragraphs retrieved from a 5M+ paragraph corpus.

Output: A predicted answer span (or 'yes'/'no' for yes/no questions) and a set of predicted supporting fact sentences.

Scoring recipe

def evaluate(pred_ans, gold_ans, pred_sup, gold_sup):
    ans_em = 1.0 if pred_ans == gold_ans else 0.0
    ans_f1 = token_f1(pred_ans, gold_ans)
    sup_em = 1.0 if set(pred_sup) == set(gold_sup) else 0.0
    sup_f1 = token_f1(' '.join(pred_sup), ' '.join(gold_sup))
    p_joint = ans_f1 * sup_f1
    r_joint = ans_f1 * sup_f1
    joint_f1 = 2 * p_joint * r_joint / (p_joint + r_joint) if (p_joint + r_joint) > 0 else 0.0
    joint_em = 1.0 if (ans_em == 1.0 and sup_em == 1.0) else 0.0
    return {'EM': ans_em, 'F1': ans_f1, 'Joint F1': joint_f1, 'Joint EM': joint_em}

Common pitfalls

• Confusing the 'distractor' setting (fixed 10 paragraphs) with the 'full wiki' setting (retrieval from 5M+ paragraphs), which drastically changes difficulty and retrieval metrics.
• Evaluating supporting facts at the paragraph level instead of the sentence level, as the dataset provides sentence-level annotations and the model predicts sentence-level probabilities.
• Misinterpreting Joint F1/EM as simple averages; they are multiplicative combinations that heavily penalize systems failing on either answer or supporting fact prediction.

Evidence (verbatim from paper)

Following previous work (Rajpurkar et al., 2016), we use exact match (EM) and F1 as two evaluation metrics. To assess the explainability of the models, we further introduce two sets of metrics involving the supporting facts. The first set focuses on evaluating the supporting facts directly, namely EM and F1 on the set of supporting fact sentences as compared to the gold set. The second set features joint metrics that combine the evaluation of answer spans and supporting facts as follows. For each example, given its precision and recall on the answer span (P^(ans), R^(ans)) and the supporting facts (P^(sup), R^(sup)), respectively, we calculate joint F1 as P^(joint) = P^(ans)P^(sup), R^(joint) = R^(ans)R^(sup), Joint F1 = 2P^(joint)R^(joint)/(P^(joint) + R^(joint)). Joint EM is 1 only if both tasks achieve an exact match and otherwise 0.

Citation

@misc{yang2018hotpotqa,
  title={HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering},
  author={Zhilin Yang et al. (2018)},
  year={2018},
  note={arXiv:1809.09600}
}

• arXiv: 1809.09600