Pubmedqa Eval

pubmedqa-eval

PubMedQA: A Dataset for Biomedical Research Question Answering — Jin et al. (2019) (arXiv:1909.06146, 2019)

What this evaluates

This benchmark evaluates a model's ability to perform biomedical research question answering by reasoning over structured scientific abstracts. It requires models to infer yes/no/maybe answers to questions derived from paper titles using only the non-conclusion sections of the abstract, without access to the final conclusion.

Datasets

• PubMedQA (PQA-L) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted labels (yes, no, or maybe) out of the total number of instances.
• macro-F1 — range: percent
  • Unweighted mean of the F1 scores calculated independently for each of the three classes (yes, no, maybe).

Input / output format

Input: A research question (derived from a title) and the context sections of a biomedical abstract (excluding the conclusion).

Output: A single categorical label: 'yes', 'no', or 'maybe'.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = ['yes', 'no', 'maybe']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(classes)
    return accuracy * 100, macro_f1 * 100

Common pitfalls

• The main evaluation uses a 'reasoning-required' setting where models must predict without seeing the abstract's conclusion; evaluating in a 'reasoning-free' setting (where the conclusion is provided) yields artificially high scores.
• The pre-training subset (PQA-A) is highly imbalanced, causing a trivial majority baseline to achieve ~92% accuracy, which can mislead comparisons if not normalized by macro-F1.
• Human performance is reported as a lower bound because it is measured with single annotators, whereas ensembles of annotators perform significantly better.

Evidence (verbatim from paper)

Under reasoning-required setting, the task becomes much harder, but it's still possible for humans to solve: a single annotator can get 78.0% accuracy and 72.2% macro-F1.

Citation

@misc{jin2019pubmedqa,
  title={PubMedQA: A Dataset for Biomedical Research Question Answering},
  author={Jin et al. (2019)},
  year={2019},
  note={arXiv:1909.06146}
}

• arXiv: 1909.06146