# Biobert Ner Eval

> Evaluates a model's ability to identify and classify biomedical entities (diseases, drugs/chemicals, genes/proteins, species) in text using transfer learning from domain-specific pre-training. It tests whether contextualized representations improve entity boundary detection and classification on small biomedical corpora. Use when the user wants to benchmark on NCBI disease, 2010 i2b2/VA, BC5CDR, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800, or asks about evaluating this task. Reports entity-level F1.

- Skill: `qhjqhj00/biobert-ner-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/biobert-ner-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/biobert-ner-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/biobert-ner-eval

---


# biobert-ner-eval

> BioBERT: a pre-trained biomedical language representation model for biomedical text mining — Lee et al. (2019) (arXiv:1901.08746, 2019)

## What this evaluates

Evaluates a model's ability to identify and classify biomedical entities (diseases, drugs/chemicals, genes/proteins, species) in text using transfer learning from domain-specific pre-training. It tests whether contextualized representations improve entity boundary detection and classification on small biomedical corpora.

## Datasets

- **NCBI disease** — total ?; splits: test (-1)
- **2010 i2b2/VA** — total ?; splits: test (-1)
- **BC5CDR** — total ?; splits: test (-1)
- **BC4CHEMD** — total ?; splits: test (-1)
- **BC2GM** — total ?; splits: test (-1)
- **JNLPBA** — total ?; splits: test (-1)
- **LINNAEUS** — total ?; splits: test (-1)
- **Species-800** — total ?; splits: test (-1)

## Metrics

- `entity-level F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall calculated at the entity span level. P = correct entities / predicted entities, R = correct entities / gold entities, F1 = 2PR/(P+R).

## Input / output format

**Input**: Raw biomedical text or sentences.

**Output**: Predicted entity spans with corresponding labels (e.g., Disease, Gene, Chemical, Species).

## Scoring recipe

```python
def compute_entity_f1(preds, gold):
  pred_set = set(preds)
  gold_set = set(gold)
  tp = len(pred_set & gold_set)
  fp = len(pred_set - gold_set)
  fn = len(gold_set - pred_set)
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
  return p, r, f1
```

## Common pitfalls

- Evaluating at token level instead of entity level as explicitly specified.
- Not averaging F1 scores across all 9 datasets for the overall comparison.
- Comparing against state-of-the-art models that use multi-task learning or different architectures, which BioBERT explicitly avoids.

## Evidence (verbatim from paper)

> For the evaluation metric, we used entity level precision, recall, and f1 score. First, we observe that BERT which was pre-trained on only the general domain corpus is quite effective. However, on average, performance of BERT was lower than that of state-of-the-art models by 2.28 in terms of F1 score. BioBERT achieves higher scores than BERT on all the datasets. On 6 out of 9 datasets, BioBERT even outperformed the current state-of-the-art models, and BioBERT (+ PubMed + PMC) outperformed the state-of-the-art models by 0.51 in terms of F1 score on average.

## Citation

```bibtex
@misc{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee et al. (2019)},
  year={2019},
  note={arXiv:1901.08746}
}
```

- arXiv: 1901.08746