qhjqhj00 clinical-relation-extraction-eval
Clinical Relation Extraction Using Transformer-based Models — Yang et al. (2021) (arXiv:2107.08957, 2021)
What this evaluates
Evaluates transformer-based models on clinical relation extraction tasks, measuring their ability to identify and classify relationships between medical entities in text. It compares general vs. clinical-pretrained architectures and binary vs. multi-class classification strategies.
Datasets
- MADE1.0 — total ?; splits: 5-fold cross-validation (-1); repo https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction
- n2c2 — total ?; splits: 5-fold cross-validation (-1); repo https://github.com/uf-hobi-informatics-lab/ClinicalTransformerRelationExtraction
Metrics
strict micro-averaged F1-score(primary) — range: [0, 1]- Harmonic mean of micro-averaged precision and recall across all relation categories. Strict matching requires exact boundary and type alignment for predicted entities and relations to count as correct.
Input / output format
Input: Clinical text passages containing entity mentions. Specific tokenization or prompt format is not detailed in the provided section.
Output: Predicted relation types between entity pairs. Exact output format is not specified, but models are fine-tuned using standard transformer architectures.
Scoring recipe
# Pseudo-code for strict micro-averaged F1
predicted_correct = 0
total_predicted = 0
total_gold = 0
for pred_rel, gold_rel in zip(predictions, gold_labels):
if pred_rel == gold_rel: # Strict match: exact entity boundaries and relation type
predicted_correct += 1
total_predicted += 1
total_gold += 1
precision = predicted_correct / total_predicted if total_predicted > 0 else 0
recall = predicted_correct / total_gold if total_gold > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Using loose entity matching instead of strict matching, which inflates scores.
- Macro-averaging across categories instead of micro-averaging, which changes the weight given to frequent vs. rare relation types.
- Not using the official 2018 n2c2 evaluation scripts, leading to inconsistent scoring.
Evidence (verbatim from paper)
Following the standard evaluation of clinical RE, we compared the performance of transformer-based models using the strict micro-averaged precision, recall, and F1-score aggregated from all relation categories. We used the official evaluation scripts provided by the 2018 n2c2 challenges to calculate these scores. The entities involved in the evaluation were from the gold standard annotations.
Citation
@misc{yang2021clinicalrelationextraction,
title={Clinical Relation Extraction Using Transformer-based Models},
author={Yang et al. (2021)},
year={2021},
note={arXiv:2107.08957}
}
- arXiv: 2107.08957