qhjqhj00 clinical-modernbert-eval
Clinical ModernBERT: An efficient and long context encoder for biomedical text — Lee et al. (2025) (arXiv:2504.03964, 2025)
What this evaluates
Evaluates a biomedical language model on short- and long-context clinical NLP tasks including classification, named entity recognition, and retrieval. It also measures pre-training masked language modeling accuracy and measures inference efficiency under varying computational loads.
Datasets
- EHR-Prediction (MIMIC-IV ED) — total 400019; splits: unspecified (-1)
- MedNER — total 3655; splits: unspecified (-1)
- Pubmed-NCT — total 221186; splits: unspecified (-1)
- PMC-Retrieval — total 167034; splits: unspecified (-1)
- i2b2 2006 — total 66034; splits: unspecified (-1)
- i2b2 2010 — total 43947; splits: unspecified (-1)
- i2b2 2012 — total 13108; splits: unspecified (-1)
- i2b2 2014 — total 83466; splits: unspecified (-1)
Metrics
top-k accuracy(primary) — range: [0, 1]- Computed over masked token positions. A prediction is correct if the ground truth token appears in the top-k most probable tokens predicted by the model. Top-1 reflects precision, top-25 captures broader lexical recall.
forward pass latency— range: milliseconds- Wall-clock time measured from input tokenization through to the final hidden state output, excluding disk/network I/O. Reported as mean across 3 runs.
Input / output format
Input: Tokenized clinical or biomedical text sequences. For pre-training, sequences are tokenized and a subset of tokens is replaced with [MASK]. For downstream tasks, inputs are padded or truncated to model-specific context limits (e.g., 512 or 8192 tokens).
Output: For MLM: predicted probability distribution over the vocabulary for masked positions. For downstream tasks: predicted class labels, entity spans, or retrieved document IDs. For efficiency: wall-clock time in milliseconds.
Scoring recipe
def compute_topk_accuracy(gold_tokens, predicted_logits, k):
correct = 0
total = 0
for gold, logits in zip(gold_tokens, predicted_logits):
top_k_indices = logits.argsort(descending=True)[:k]
if gold in top_k_indices:
correct += 1
total += 1
return correct / total if total > 0 else 0.0
Common pitfalls
- Pre-training MLM evaluation uses top-k accuracy, but downstream task metrics (e.g., F1 for NER, accuracy for classification) are not explicitly defined in this section and follow standard task conventions.
- Efficiency benchmarking uses synthetic clinical text fixed at 512 tokens and excludes I/O; real-world clinical inference latency will be higher and variable.
- Long-context tasks (i2b2 series) exceed standard 512-token limits, requiring models to handle sequences up to ~14k tokens without truncation.
Evidence (verbatim from paper)
To quantify performance, we compute top-k accuracy over masked positions, where a prediction is considered correct if the ground truth token appears in the top-k most probable tokens predicted by the model. Specifically, for each masked token $x_{i}$, we sort the predicted distribution and check whether $x_{i}$ lies in the top $k$ logits.
Citation
@misc{lee2025clinicalmodernbert,
title={Clinical ModernBERT: An efficient and long context encoder for biomedical text},
author={Lee et al. (2025)},
year={2025},
note={arXiv:2504.03964}
}
- arXiv: 2504.03964