Biopulse QA Eval

biopulse-qa-eval

BioPulse-QA: A Dynamic Biomedical Question-Answering Benchmark for Evaluating Factuality, Robustness, and Bias in Large Language Models — Bhattarai et al. (2026) (arXiv:2601.12632, 2026)

What this evaluates

This benchmark evaluates large language models on biomedical question-answering, specifically probing their factuality, robustness to linguistic variations (paraphrasing and typos), and susceptibility to demographic bias (age and gender). It distinguishes between extractive and abstractive reasoning capabilities using expert-verified QA pairs from clinical documents.

Datasets

• BioPulse-QA — total 2280; splits: test (2280)

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score (harmonic mean of precision and recall). The paper also reports a 'Relaxed F1' variant that tolerates minor formatting or whitespace differences while requiring exact match of core factual content.

Input / output format

Input: A context snippet from a biomedical document (drug label, clinical trial protocol, or clinical guideline) followed by a question.

Output: Free-text answer string (either an exact span from the text for extractive tasks or a generated sentence for abstractive tasks).

Scoring recipe

def compute_f1(pred, gold):
    pred_tokens = set(normalize(pred))
    gold_tokens = set(normalize(gold))
    intersection = pred_tokens & gold_tokens
    precision = len(intersection) / len(pred_tokens) if pred_tokens else 0
    recall = len(intersection) / len(gold_tokens) if gold_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• High performance on abstractive QA does not guarantee accuracy on extractive QA; evaluating both is necessary to avoid masking weaknesses.
• Automated metrics like Relaxed F1 can overestimate correctness; manual error analysis is required to catch factual incorrectness and incompleteness.
• Robustness testing must include semantic paraphrasing, as models show high resilience to surface-level typos but significant sensitivity to rewording.

Evidence (verbatim from paper)

For abstractive QA, GPT-o1 achieved high F1 performance on drug labels, (0.98), and Gemini-2.0-Flash achieved high performance on clinical trials (0.65) and clinical guidelines (0.93) (Figure 4). Llama-3.1-8B underperformed with a low F1 score on drug labels (0.85), clinical trials (0.62), and clinical guidelines (0.69).

Citation

@misc{bhattarai2026biopulseqa,
  title={BioPulse-QA: A Dynamic Biomedical Question-Answering Benchmark for Evaluating Factuality, Robustness, and Bias in Large Language Models},
  author={Bhattarai et al. (2026)},
  year={2026},
  note={arXiv:2601.12632}
}

• arXiv: 2601.12632