Blurb Eval

blurb-eval

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing — Gu et al. (2020) (arXiv:2007.15779, 2020)

What this evaluates

Evaluates biomedical language models on a comprehensive suite of downstream NLP tasks, including named entity recognition, relation extraction, sentence similarity, document classification, and question answering. It measures how well domain-specific pretraining transfers to specialized clinical and biomedical text understanding.

Datasets

• BLURB — total ?; splits: test (-1)

Metrics

• BLURB score (primary) — range: percent
  • Macro average of the average test results across six downstream tasks: NER, PICO, relation extraction, sentence similarity, document classification, and question answering.
• Task-specific F1 / Accuracy — range: percent
  • Entity-level F1 for NER tasks; Micro F1 for relation extraction tasks; and standard accuracy or similarity scores for sentence similarity, document classification, and QA tasks.

Input / output format

Input: Text snippets from biomedical literature (abstracts or full texts) formatted for specific downstream tasks such as entity recognition, relation extraction, sentence similarity, document classification, and question answering.

Output: Predicted labels or scores per instance (e.g., entity tags, relation types, similarity scores, class labels, or answer probabilities).

Scoring recipe

def compute_blurb_score(predictions, golds):
    task_scores = []
    for task in ['NER', 'PICO', 'RE', 'SS', 'DC', 'QA']:
        task_scores.append(average_test_results(task, predictions, golds))
    return mean(task_scores)

# Task-specific metrics use standard F1 or accuracy depending on the task type.

Common pitfalls

• Using out-of-domain vocabulary or mixed-domain pretraining significantly degrades performance on biomedical tasks compared to domain-specific pretraining from scratch.
• Complex BIO tagging schemes for NER do not improve over simple IO tagging when using self-attention models like BERT, contrary to traditional sequential models.
• Adversarial pretraining and adding full-text PMC data can degrade performance unless training is extended significantly (e.g., 60% longer).

Evidence (verbatim from paper)

The BLURB score is the macro average of average test results for each of the six tasks (NER, PICO, relation extraction, sentence similarity, document classification, question answering). See Table 3 for the evaluation metric used in each task.

Citation

@misc{gu2020domain,
  title={Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing},
  author={Gu et al. (2020)},
  year={2020},
  note={arXiv:2007.15779}
}

• arXiv: 2007.15779