Qasper Eval

qasper-eval

A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers — Dasigi et al. (2021) (arXiv:2105.03011, 2021)

What this evaluates

This benchmark evaluates a model's ability to read and reason over full academic research papers to answer information-seeking questions and to identify the specific paragraphs that contain the supporting evidence. It probes document-level comprehension, multi-paragraph reasoning, and handling of diverse answer types including extractive spans, abstractive summaries, yes/no, and unanswerable cases.

Datasets

• QASPER — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• Answer F1 (primary) — range: [0, 1]
  • Token-level F1 score computed between the predicted answer and the gold answer. It measures exact token overlap regardless of answer type (extractive, abstractive, yes/no, or unanswerable).
• Evidence F1 — range: [0, 1]
  • Paragraph-level F1 score measuring the overlap between the set of predicted evidence paragraphs and the set of gold evidence paragraphs. Precision and recall are computed over paragraph sets, and F1 is the harmonic mean.

Input / output format

Input: A question string concatenated with a document context (e.g., question only, question + abstract, question + introduction, or question + full paper text).

Output: A predicted answer (extractive span, abstractive text, yes/no, or unanswerable) and/or a set of selected evidence paragraphs from the input document.

Scoring recipe

def compute_f1(pred, gold):
    pred_tokens = set(pred.lower().split())
    gold_tokens = set(gold.lower().split())
    if not pred_tokens or not gold_tokens:
        return 0.0
    precision = len(pred_tokens & gold_tokens) / len(pred_tokens)
    recall = len(pred_tokens & gold_tokens) / len(gold_tokens)
    return 2 * precision * recall / (precision + recall)

def compute_evidence_f1(pred_paras, gold_paras):
    pred_set = set(p.strip() for p in pred_paras)
    gold_set = set(p.strip() for p in gold_paras)
    if not pred_set or not gold_set:
        return 0.0
    precision = len(pred_set & gold_set) / len(pred_set)
    recall = len(pred_set & gold_set) / len(gold_set)
    return 2 * precision * recall / (precision + recall)

Common pitfalls

• The dataset explicitly excludes questions requiring figures or tables (13% of items), so models must be evaluated only on text-answering questions to match the reported protocol.
• Human performance is estimated as a lower bound by treating one human annotation as a prediction and scoring against the rest, rather than using a single gold standard.
• Answer F1 is computed at the token level, not character level or exact match, which heavily penalizes minor wording differences in abstractive answers.

Evidence (verbatim from paper)

Table 2: LED-base and lower-bound human performance on answering questions in QASPER, measured in Answer F1.

Citation

@misc{dasigi2021qasper,
  title={A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers},
  author={Dasigi et al. (2021)},
  year={2021},
  note={arXiv:2105.03011}
}

• arXiv: 2105.03011