Biobert Ner Eval

biobert-ner-eval

BioBERT: a pre-trained biomedical language representation model for biomedical text mining — Lee et al. (2019) (arXiv:1901.08746, 2019)

What this evaluates

Evaluates a model's ability to identify and classify biomedical entities (diseases, drugs/chemicals, genes/proteins, species) in text using transfer learning from domain-specific pre-training. It tests whether contextualized representations improve entity boundary detection and classification on small biomedical corpora.

Datasets

• NCBI disease — total ?; splits: test (-1)
• 2010 i2b2/VA — total ?; splits: test (-1)
• BC5CDR — total ?; splits: test (-1)
• BC4CHEMD — total ?; splits: test (-1)
• BC2GM — total ?; splits: test (-1)
• JNLPBA — total ?; splits: test (-1)
• LINNAEUS — total ?; splits: test (-1)
• Species-800 — total ?; splits: test (-1)

Metrics

• entity-level F1 (primary) — range: percent
  • Harmonic mean of precision and recall calculated at the entity span level. P = correct entities / predicted entities, R = correct entities / gold entities, F1 = 2PR/(P+R).

Input / output format

Input: Raw biomedical text or sentences.

Output: Predicted entity spans with corresponding labels (e.g., Disease, Gene, Chemical, Species).

Scoring recipe

def compute_entity_f1(preds, gold):
  pred_set = set(preds)
  gold_set = set(gold)
  tp = len(pred_set & gold_set)
  fp = len(pred_set - gold_set)
  fn = len(gold_set - pred_set)
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
  return p, r, f1

Common pitfalls

• Evaluating at token level instead of entity level as explicitly specified.
• Not averaging F1 scores across all 9 datasets for the overall comparison.
• Comparing against state-of-the-art models that use multi-task learning or different architectures, which BioBERT explicitly avoids.

Evidence (verbatim from paper)

For the evaluation metric, we used entity level precision, recall, and f1 score. First, we observe that BERT which was pre-trained on only the general domain corpus is quite effective. However, on average, performance of BERT was lower than that of state-of-the-art models by 2.28 in terms of F1 score. BioBERT achieves higher scores than BERT on all the datasets. On 6 out of 9 datasets, BioBERT even outperformed the current state-of-the-art models, and BioBERT (+ PubMed + PMC) outperformed the state-of-the-art models by 0.51 in terms of F1 score on average.

Citation

@misc{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee et al. (2019)},
  year={2019},
  note={arXiv:1901.08746}
}

• arXiv: 1901.08746