Medbert De Med Bench Eval

medbert-de-med-bench-eval

MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain — Bressem et al. (2023) (arXiv:2303.08179, 2023)

What this evaluates

Evaluates German medical language models on classification and named entity recognition tasks across radiology reports, clinical discharge notes, surgery reports, and public medical/general benchmarks.

Datasets

• Chest CT — total 2000; splits: train (1000), val (500), test (500)
• Chest X-Ray — total 2000; splits: train (1000), val (500), test (500)
• ICD-10 code classification on discharge notes — total 2000; splits: train (1000), val (500), test (500)
• OPS code classification on discharge notes — total 2000; splits: train (1000), val (500), test (500)
• OPS code classification on surgery reports — total 2000; splits: train (1000), val (500), test (500)
• GermEval-18 — total ?; splits: test (-1)
• Wrist NER — total ?; splits: test (-1)
• GraSCCo — total ?; splits: test (-1)
• GGPOnc — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Computed per class or globally depending on the task.
• Macro F1 — range: [0, 1]
  • Unweighted mean of the F1 scores for each class, treating all classes equally regardless of support.
• Micro F1 — range: [0, 1]
  • F1 score calculated globally by counting total true positives, false negatives, and false positives across all classes.
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive observations to the total predicted positives.
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive observations to all observations in the actual class.
• Token-level F1 (F1tok) — range: [0, 1]
  • F1 score computed at the token level for NER tasks, aggregating predictions across all tokens in the sequence.

Input / output format

Input: German medical text documents (e.g., radiology reports, discharge summaries, surgery reports, clinical guidelines, or general domain text).

Output: Predicted class labels for classification tasks; predicted entity tags for NER tasks.

Scoring recipe

def compute_classification_metrics(y_true, y_pred, y_prob=None):
    precision = precision_score(y_true, y_pred, average='macro')
    recall = recall_score(y_true, y_pred, average='macro')
    f1 = f1_score(y_true, y_pred, average='macro')
    auc = roc_auc_score(y_true, y_prob, multi_class='ovr') if y_prob is not None else None
    return {'Precision': precision, 'Recall': recall, 'Macro F1': f1, 'AUROC': auc}

def compute_ner_metrics(y_true_tokens, y_pred_tokens):
    # Token-level F1/Precision/Recall computed by flattening sequences and ignoring padding
    prec = precision_score(y_true_tokens, y_pred_tokens, average='macro')
    rec = recall_score(y_true_tokens, y_pred_tokens, average='macro')
    f1 = f1_score(y_true_tokens, y_pred_tokens, average='macro')
    return {'Prec': prec, 'Rec': rec, 'F1tok': f1}

Common pitfalls

• Private benchmarks (Chest CT, X-Ray, ICD-10, OPS, Wrist NER) are not publicly available, limiting independent reproduction.
• Token-level vs. document-level metrics are reported separately but the text does not specify how token-level AUROC/F1 is aggregated across sequences.
• Deduplication variants are evaluated but show negligible performance differences, making it unclear if data cleaning pipelines are standardized.

Evidence (verbatim from paper)

In the chest x-ray task, we found that the two best performing models were our own pre-trained BERT models. Our model trained on the corpus with duplicates removed (medBERT.dededup) achieves a slightly better performance with an average AUROC of 83.65 compared to 83.42 of the model trained on the whole corpus (medBERT.de).

Citation

@misc{bressem2023medbertde,
  title={MEDBERT.de: A Comprehensive German BERT Model for the Medical Domain},
  author={Bressem et al. (2023)},
  year={2023},
  note={arXiv:2303.08179}
}

• arXiv: 2303.08179