Beerqa Eval

beerqa-eval

Answering Open-Domain Questions of Varying Reasoning Steps from Text — Peng Qi et al. (2020) (arXiv:2010.12527, 2020)

What this evaluates

Evaluates open-domain question answering systems on their ability to retrieve and synthesize information across varying numbers of reasoning steps (single-hop to three-hop) without relying on structured metadata or predefined hop counts.

Datasets

• SQuAD Open — total 75841; splits: train (59285), dev (8132), test (8424)
• HotpotQA — total 86725; splits: train (74758), dev (5989), test (5978)
• BeerQA — total 163096; splits: train (134043), dev (14121), test (14932); repo https://github.com/beerqa/IRRR

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • 1 if the predicted answer string exactly matches the gold answer string, else 0.
• unigram F1 — range: [0, 1]
  • Token-level unigram F1 score computed between the predicted answer and the gold answer.

Input / output format

Input: Question text and retrieved Wikipedia paragraphs/pages.

Output: Answer string.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = []
    for p, g in zip(predictions, golds):
        p_tokens = set(p.lower().split())
        g_tokens = set(g.lower().split())
        if not p_tokens or not g_tokens: continue
        prec = len(p_tokens & g_tokens) / len(p_tokens)
        rec = len(p_tokens & g_tokens) / len(g_tokens)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}

Common pitfalls

• Naively merging corpora causes repeated or contradicting information, leading models to use corpus style as a shortcut or generate false answers.
• Models may fail to generalize to unseen multi-hop reasoning steps if trained only on fixed-hop datasets.
• Corpora are mapped to a newer Wikipedia dump (Aug 2020), requiring careful filtering of removed or edited pages to maintain answer validity.

Evidence (verbatim from paper)

For all benchmark datasets, we report standard answer exact match (EM) and unigram F1 metrics.

Citation

@misc{qi2020answering,
  title={Answering Open-Domain Questions of Varying Reasoning Steps from Text},
  author={Peng Qi et al. (2020)},
  year={2020},
  note={arXiv:2010.12527}
}

• arXiv: 2010.12527