Morehopqa Eval

morehopqa-eval

MoreHopQA: More Than Multi-hop Reasoning — Schnitzler et al. (2024) (arXiv:2406.13397, 2024)

What this evaluates

Evaluates multi-step reasoning capabilities beyond simple information extraction in question answering. It probes models' ability to perform arithmetic, commonsense, and symbolic reasoning by extending standard multi-hop questions with additional reasoning layers.

Datasets

• MoreHopQA — total 1118; splits: test (1118); repo https://github.com/Alab-NII/morehopqa

Metrics

• EM (primary) — range: [0, 1]
  • Exact Match score: 1 if the postprocessed model-generated answer exactly matches the preprocessed ground-truth answer, 0 otherwise. Averaged over the dataset.
• F1 — range: [0, 1]
  • Token-level F1 score computed between the postprocessed model-generated answer and the preprocessed ground-truth answer.

Input / output format

Input: Instruction prompt containing the question, supporting context, and optional few-shot examples. Models are instructed to output the final answer enclosed in tags.

Output: Text string enclosed between and tags.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    em_scores, f1_scores = [], []
    for pred, gt in zip(predictions, ground_truths):
        pred_str = extract_between(pred, '<answer>', '</answer>')
        pred_proc = postprocess_to_string(pred_str)
        gt_proc = preprocess_string(gt)
        em_scores.append(1.0 if pred_proc == gt_proc else 0.0)
        f1_scores.append(token_f1(pred_proc, gt_proc))
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)

Common pitfalls

• Models often exhibit shortcut reasoning (answering the initial question correctly but failing sub-questions), which requires separate tracking rather than relying solely on overall accuracy.
• The evaluation mandates specific postprocessing (NER/datatype conversion) before comparison; skipping this step yields artificially low EM/F1 scores.
• Gemma-7B frequently outputs refusal strings like 'I cannot answer', which must be filtered or handled to prevent skewing the average performance.

Evidence (verbatim from paper)

We then attempt to convert this string into the respective built-in python datatype for the answer type, either directly or with the help of Named Entity Recognition, and convert it back to a default string representation. We then report the EM and F1 scores on the tokens between the preprocessed ground-truth answer and the postprocessed model-generated answer.

Citation

@misc{schnitzler2024morehopqa,
  title={MoreHopQA: More Than Multi-hop Reasoning},
  author={Schnitzler et al. (2024)},
  year={2024},
  note={arXiv:2406.13397}
}

• arXiv: 2406.13397