Blue Eval

blue-eval

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets — Peng et al. (2019) (arXiv:1906.05474, 2019)

What this evaluates

Evaluates the cross-domain generalization and transfer learning capabilities of pre-trained language models across ten diverse biomedical and clinical NLP tasks. It probes sentence similarity, named entity recognition, relation extraction, document classification, and natural language inference to measure how well domain-specific pre-training captures clinical and biomedical semantics.

Datasets

• MedSTS — total 1068; splits: train (675), dev (75), test (318)
• BIOSSES — total 100; splits: train (64), dev (16), test (20)
• BC5CDR-disease — total 12850; splits: train (4182), dev (4244), test (4424)
• BC5CDR-chemical — total 15935; splits: train (5203), dev (5347), test (5385)
• ShARe/CLEFE — total 10898; splits: train (4628), dev (1075), test (5195)
• DDI — total 4920; splits: train (2937), dev (1004), test (979)
• ChemProt — total 10028; splits: train (4154), dev (2416), test (3458)
• i2b2 2010 — total 9414; splits: train (3110), dev (11), test (6293)
• HoC — total 1580; splits: train (1108), dev (157), test (315)
• MedNLI — total 14049; splits: train (11232), dev (1395), test (1422)

Metrics

• Total Score (Macro-average) (primary) — range: other
  • Macro-average of all task-specific metrics (F1-scores and Pearson correlation coefficients) across the ten datasets.
• Pearson correlation — range: [-1, 1]
  • Linear correlation coefficient between predicted and gold similarity scores for sentence pairs.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall. Evaluated using strict matching for NER, micro-averaging for relation extraction, and example-based averaging for document classification.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified premise-hypothesis pairs in the inference task.

Input / output format

Input: Task-dependent: sentence pairs for similarity and inference, tokenized text with entity mentions for NER, sentences with entity pairs for relation extraction, and full documents for classification.

Output: Task-dependent: continuous similarity scores (0–5), BIO/IO entity span labels, relation type labels between entity pairs, multi-label document categories, or entailment/contradiction/neutral labels.

Scoring recipe

def compute_total_score(predictions, golds):
    scores = []
    for task in tasks:
        if task == 'similarity':
            scores.append(pearson_corr(predictions[task], golds[task]))
        elif task == 'ner':
            scores.append(f1_score(predictions[task], golds[task], average='strict'))
        elif task == 'relation_extraction':
            scores.append(f1_score(predictions[task], golds[task], average='micro'))
        elif task == 'document_classification':
            scores.append(f1_score(predictions[task], golds[task], average='example'))
        elif task == 'inference':
            scores.append(accuracy_score(predictions[task], golds[task]))
    return mean(scores)

Common pitfalls

• BIOSSES lacks official train/test splits, requiring a manual 80/20 random split that may introduce variance.
• The benchmark uses three different F1 variants (strict, micro, example-based) across tasks, making direct cross-task comparison of raw scores invalid.
• The headline 'Total Score' macro-averages Pearson correlation (range [-1,1]) with F1/accuracy (range [0,1]), which can mask poor performance on specific tasks.

Evidence (verbatim from paper)

Following the practice in Wang et al. (2018a) and Lee et al. (2019), we use a macro-average of F1-scores and Pearson scores to determine a system’s position.

Citation

@misc{peng2019blue,
  title={Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets},
  author={Peng et al. (2019)},
  year={2019},
  note={arXiv:1906.05474}
}

• arXiv: 1906.05474