Multi Hop QA Eval

multi-hop-qa-eval

Answering Questions by Meta-Reasoning over Multiple Chains of Thought — Yoran et al. (2023) (arXiv:2304.13007, 2023)

What this evaluates

Evaluates multi-hop question answering capabilities across diverse reasoning types, including implicit commonsense/arithmetic reasoning, explicit composition/comparison, and fact verification. It tests the model's ability to synthesize information from retrieved evidence and generate step-by-step explanations.

Datasets

• STRATEGYQA — total 1000; splits: dev (1000), test (490)
• FERMI — total 286; splits: dev (286), test (558)
• QUARTZ — total 374; splits: eval (374)
• HOTPOTQA — total 500; splits: eval (500)
• 2WIKIMQA — total 500; splits: eval (500)
• BAMBOOGLE — total 120; splits: eval (120)
• FEVEROUS — total 500; splits: eval (500)

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score between the predicted answer and the gold answer. Used for all explicit reasoning datasets.
• exact-match — range: [0, 1]
  • Exact string match between the predicted answer and the gold answer. Used for binary-choice datasets.
• order-of-magnitude — range: [0, 1]
  • Official FERMI evaluation metric that checks if the predicted answer falls within an order of magnitude of the gold answer.

Input / output format

Input: Question and retrieved context (top-1 evidence sentence or list from Wikipedia via Google Search or ColBERTv2), formatted with in-context exemplars showing step-by-step reasoning.

Output: Step-by-step reasoning followed by a final answer. For binary-choice datasets, a Yes/No answer. 'Unknown' if the model is unsure.

Scoring recipe

def evaluate(predictions, golds, dataset_type):
    if dataset_type == 'explicit':
        return f1_score(predictions, golds)
    elif dataset_type == 'binary':
        return exact_match(predictions, golds)
    elif dataset_type == 'fermi':
        return order_of_magnitude_accuracy(predictions, golds)
    return None

Common pitfalls

• Using the development set instead of the official test set for STRATEGYQA and FERMI when comparing to reported test results.
• Applying exact-match scoring to open-ended explicit reasoning datasets instead of token-level F1.
• Using standard exact-match or F1 for FERMI instead of the official order-of-magnitude evaluation.

Evidence (verbatim from paper)

For evaluation, we use F1 to compare predicted and gold answers for all explicit reasoning datasets and exact-match for the binary-choice datasets. In FERMI, we use the official order-of-magnitude evaluation by Kalyan et al. (2021).

Citation

@misc{yoran2023answering,
  title={Answering Questions by Meta-Reasoning over Multiple Chains of Thought},
  author={Yoran et al. (2023)},
  year={2023},
  note={arXiv:2304.13007}
}

• arXiv: 2304.13007