Bioclinical Modernbert Eval

bioclinical-modernbert-eval

BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP — Sounack et al. (2025) (arXiv:2506.10896, 2025)

What this evaluates

Evaluates long-context clinical NLP encoders on biomedical entity recognition, clinical text classification, and demographic information extraction. Probes the model's ability to process full-length clinical notes (up to 8,192 tokens) and retain domain-specific knowledge without truncation.

Datasets

• ChemProt — total ?; splits: test (-1)
• Phenotype — total ?; splits: test (-1)
• Social History — total ?; splits: test (-1)
• DEID — total ?; splits: test (-1)
• COS — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: percent
  • Harmonic mean of precision and recall. For classification tasks, macro-averaged across all classes. For NER tasks, entity-level exact match F1 is used.

Input / output format

Input: Clinical text sequences (PubMed abstracts, clinical notes, social history records) tokenized with a 50,368-token vocabulary, processed at fixed lengths (512, 4096, or 8192 tokens) or as variable-length sequences.

Output: Class labels for classification tasks (e.g., chemical-protein relations, phenotype categories) or token-level entity tags for NER tasks (e.g., social history entities, demographic identifiers, clinical observations).

Scoring recipe

def compute_f1(predictions, gold_labels):
    # Classification: macro-averaged F1
    # NER: entity-level exact match F1
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == g and p != 'O')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p != g and p != 'O')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p != g and g != 'O')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100  # Return as percentage

Common pitfalls

• Confusing fixed-length vs variable-length inference speed benchmarks; some models degrade significantly on variable-length sequences due to lack of unpadding.
• Using short-context truncation (e.g., 512 tokens) for clinical notes, which loses critical information compared to the model's designed 8,192-token context window.
• Evaluating Phase 1 vs Phase 2 checkpoints without clarifying that Phase 2 includes clinical specialization data, which can artificially inflate clinical task scores.

Evidence (verbatim from paper)

For classification tasks, BioClinical ModernBERT large achieves state-of-the-art results with an F1 score of 90.8% on ChemProt and 60.8% on Phenotype. The base model also outperforms all other base models, achieving 89.9% on ChemProt and 58.1% on Phenotype. In named entity recognition, the base model achieves state-of-the-art performance on Social History and outperforms other base models on DEID.

Citation

@misc{sounack2025bioclinicalmodernbert,
  title={BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP},
  author={Sounack et al. (2025)},
  year={2025},
  note={arXiv:2506.10896}
}

• arXiv: 2506.10896