Medication Information Extraction Eval

medication-information-extraction-eval

Contextualized Medication Information Extraction Using Transformer-based Deep Learning Architectures — Chen et al. (2023) (arXiv:2303.08259, 2023)

What this evaluates

Evaluates transformer-based models on identifying medication mentions, classifying medication-related events, and determining contextual attributes (e.g., negation, temporality, certainty) from clinical narratives.

Datasets

• Challenge test dataset — total ?; splits: test (-1)

Metrics

• micro-averaged F1-score (primary) — range: [0, 1]
  • Standard F1-score calculated by aggregating true positives, false positives, and false negatives globally across all instances and classes (micro-averaging), then computing precision and recall. F1 = 2 * (precision * recall) / (precision + recall).
• macro-averaged F1-score — range: [0, 1]
  • Standard F1-score calculated by independently computing precision and recall for each class, then taking the unweighted mean across all classes (macro-averaging).

Input / output format

Input: Clinical narrative text containing medication mentions and contextual information.

Output: Predicted medication mentions, event classifications, and contextual attribute labels.

Scoring recipe

def calculate_micro_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    fp = sum(1 for p, g in zip(predictions, gold) if p != g and p in set(gold))
    fn = sum(1 for p, g in zip(predictions, gold) if p != g and g not in set(predictions))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
# Note: Paper specifies using the official challenge evaluation script for exact token/entity matching.

Common pitfalls

• Micro-averaging can mask poor performance on rare medication classes, while macro-averaging treats all classes equally regardless of frequency.
• Context classification involves multiple overlapping attributes (negation, temporality, etc.), requiring careful handling of multi-label vs. single-label evaluation conventions.
• Reliance on a challenge-specific official script may obscure exact tokenization or entity-matching rules used for precision/recall calculation.

Evidence (verbatim from paper)

Following the evaluation metrics used in this challenge, the micro- and macro-averaged precision, recall, and F1-score were used to evaluate medication extraction; the micro-averaged precision, recall and F1-score were used to evaluate the event classification and the contextual classification. All evaluation scores were calculated using the official evaluation script and the test data set provided in this challenge.

Citation

@misc{chen2023contextualized,
  title={Contextualized Medication Information Extraction Using Transformer-based Deep Learning Architectures},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2303.08259}
}

• arXiv: 2303.08259