Dutch Ade Corpus Eval

dutch-ade-corpus-eval

Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study — Murphy et al. (2025) (arXiv:2507.19396, 2025)

What this evaluates

This benchmark evaluates transformer and Bi-LSTM models for detecting adverse drug events (ADEs) in Dutch clinical free text. It probes named entity recognition for drugs and disorders, relation classification for ADE and prescribing indication pairs, and document-level ADE detection. The protocol emphasizes handling class imbalance and evaluating performance across strict/lenient entity matching and single vs. grouped ADE relations.

Datasets

• Dutch ADE corpus — total ?; splits: train (-1), val (-1), test (-1)
• ICU AKI corpus — total ?; splits: test (-1)
• WINGS corpus — total ?; splits: test (-1)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • F1 = 2 * (precision * recall) / (precision + recall). Macro-averaging computes F1 for each class independently and averages them equally, treating each class as equally important regardless of support.
• micro-F1 — range: [0, 1]
  • Computes global precision and recall across all instances before calculating F1, giving equal weight to every prediction rather than each class.
• F2 score — range: [0, 1]
  • F-beta score with beta=2, weighting recall twice as heavily as precision to prioritize catching all ADE mentions over precision.

Input / output format

Input: Clinical sentences or full notes containing drug and disorder mentions. For relation classification, inputs include contextualized embeddings for the CLS token and two candidate entities (≤4 sentences apart), plus entity label probability vectors.

Output: Per-token BIO probability distributions (B/drug, I/drug, B/disorder, I/disorder, O). Binary classification probabilities for ADE vs non-ADE and prescribing indication vs no prescribing indication for each entity pair. Document-level binary ADE presence label.

Scoring recipe

def compute_f1(y_true, y_pred, average='macro', matching='strict'):
    if matching == 'strict':
        y_true, y_pred = exact_span_align(y_true, y_pred)
    elif matching == 'lenient':
        y_true, y_pred = token_overlap_align(y_true, y_pred)
    return f1_score(y_true, y_pred, average=average, zero_division=0)
# Threshold selection: search PR curve on validation set to find threshold maximizing F2

Common pitfalls

• Micro-averaging F1 can mask poor performance on minority classes in highly imbalanced clinical datasets.
• Strict vs. lenient entity matching drastically changes NER scores; exact span matches are required for strict, while token overlap suffices for lenient.
• Evaluating at the single ADE relation level versus the unique ADE group level (easy vs. hard setting) yields fundamentally different clinical utility assessments.

Evidence (verbatim from paper)

For all tasks, we calculated micro- and macro-averaged F1 score using the python scikit-learn library [32]. Macro-averaging shows the performance across classes treating each class as equally important, whereas micro-averaging gives equal weight to every instance and shows average performance across all predictions. Micro-averaging can hide low performance on minority classes for classification problems with class imbalance.

Citation

@misc{murphy2025detection,
  title={Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study},
  author={Murphy et al. (2025)},
  year={2025},
  note={arXiv:2507.19396}
}

• arXiv: 2507.19396