Biobert QA Eval

biobert-qa-eval

BioBERT: a pre-trained biomedical language representation model for biomedical text mining — Lee et al. (2019) (arXiv:1901.08746, 2019)

What this evaluates

Evaluates factoid question answering performance on small biomedical datasets, testing transfer learning effectiveness from domain-specific pre-training. It measures how well the model retrieves exact or lenient answers to biomedical queries.

Datasets

• BioASQ 4b — total ?; splits: test (-1)
• BioASQ 5b — total ?; splits: test (-1)
• BioASQ 6b — total ?; splits: test (-1)

Metrics

• Strict Accuracy — range: percent
  • Percentage of questions where the predicted answer exactly matches the gold answer.
• Lenient Accuracy — range: percent
  • Percentage of questions where the predicted answer matches the gold answer allowing for minor variations or synonyms.
• Mean Reciprocal Rank (MRR) (primary) — range: [0, 1]
  • Average of the reciprocal ranks of the first correct answer across all questions. Rank is the position of the first correct answer in the ranked list.

Input / output format

Input: Factoid questions converted to SQuAD format.

Output: Predicted answer spans or text.

Scoring recipe

def compute_qa_metrics(preds, gold):
  strict_correct = sum(1 for p, g in zip(preds, gold) if p == g)
  lenient_correct = sum(1 for p, g in zip(preds, gold) if is_lenient_match(p, g))
  ranks = [1/(i+1) for i, (p, g) in enumerate(zip(preds, gold)) if is_match(p, g)]
  mrr = sum(ranks) / len(ranks) if ranks else 0
  return strict_correct/len(preds), lenient_correct/len(preds), mrr

Common pitfalls

• Averaging best scores from multiple different models per batch instead of using a single model on every batch, which the paper explicitly contrasts with state-of-the-art reporting.
• Ignoring the small dataset size (few hundreds of samples) which makes transfer learning critical and causes high variance.
• Not converting BioASQ factoid datasets to SQuAD format as specified in the experimental setup.

Evidence (verbatim from paper)

The QA results are shown in Table 6. Note that transfer learning is quite important in biomedical QA as the average size of datasets is very small (only few hundreds of samples). BERT outperforms the state-of-the-art models by 3 in terms of MRR on average. BioBERT (+ PubMed + PMC) significantly outperforms BERT and the state-of-the-art models, and obtained a Strict Accuracy of 37.87, Lenient Accuracy of 51.43, and a Mean Reciprocal Rank score of 43.29 on average. On all the biomedical QA datasets, BioBERT achieved new state-of-the-art performance in terms of MRR.

Citation

@misc{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee et al. (2019)},
  year={2019},
  note={arXiv:1901.08746}
}

• arXiv: 1901.08746