Biobert Re Eval

biobert-re-eval

BioBERT: a pre-trained biomedical language representation model for biomedical text mining — Lee et al. (2019) (arXiv:1901.08746, 2019)

What this evaluates

Tests a model's capability to extract biomedical relations (gene-disease, gene-chemical) from text using minimal task-specific modifications. It probes whether domain-specific pre-training improves relation classification on small-scale biomedical corpora.

Datasets

• GAD — total ?; splits: test (-1)
• EU-ADR — total ?; splits: test (-1)
• CHEMPROT — total ?; splits: test (-1)

Metrics

• entity-level F1 (primary) — range: percent
  • Harmonic mean of precision and recall calculated at the entity pair level. P = correct relations / predicted relations, R = correct relations / gold relations, F1 = 2PR/(P+R).

Input / output format

Input: Biomedical text containing entity pairs.

Output: Predicted relation labels between entity pairs.

Scoring recipe

def compute_entity_f1(preds, gold):
  pred_set = set(preds)
  gold_set = set(gold)
  tp = len(pred_set & gold_set)
  fp = len(pred_set - gold_set)
  fn = len(gold_set - pred_set)
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
  return p, r, f1

Common pitfalls

• Using complex linguistic features or multi-task learning architectures that differ from the single-architecture setup used for BERT/BioBERT.
• Not reporting entity-level metrics as specified in the protocol.
• Averaging results across datasets without weighting or reporting per-dataset scores separately.

Evidence (verbatim from paper)

The RE results are shown in Table 5. Unlike the NER results, BERT achieves better performance than that of the state-of-the-art models, which demonstrates its effectiveness in RE. On average, BioBERT (+ PubMed + PMC) outperformed the state-of-the-art models by 3.49 in terms of F1 score. On 2 out of 3 biomedical RE datasets, BioBERT achieved state-of-the-art performance in terms of F1 score. Considering the complexity of the state-of-the-art models which use numerous linguistic features (Bhasuran and Natarajan, 2018), the results are very promising.

Citation

@misc{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee et al. (2019)},
  year={2019},
  note={arXiv:1901.08746}
}

• arXiv: 1901.08746