N2c2 Concept Relation Eval

n2c2-concept-relation-eval

Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension — Peng et al. (2023) (arXiv:2303.08262, 2023)

What this evaluates

Evaluates clinical NLP models on extracting medical concepts and their relations from clinical notes. It probes the model's ability to handle nested/overlapped concepts and assesses cross-institutional generalization across different benchmark years.

Datasets

• n2c2 2018 — total ?; splits: test (-1)
• n2c2 2022 — total ?; splits: test (-1)
• n2c2 cross-institution (MIMIC-train/UW-test) — total ?; splits: train (-1), test (-1)

Metrics

• strict micro-averaged F1-score (primary) — range: [0, 1]
  • Micro-averaged F1 computed over all concept and relation categories. Strict matching requires exact span boundary and type alignment between prediction and gold. Calculated using official 2018 n2c2 challenge evaluation scripts.
• lenient micro-averaged F1-score — range: [0, 1]
  • Micro-averaged F1 computed over all concept and relation categories. Lenient matching allows partial boundary overlap for nested/overlapped concepts.

Input / output format

Input: Clinical text passages formatted as machine reading comprehension (MRC) prompts/questions.

Output: Extracted clinical concepts and relations, evaluated against gold annotations.

Scoring recipe

tp, fp, fn = 0, 0, 0
for pred_set, gold_set in zip(predictions, golds):
    for pred in pred_set:
        if pred in gold_set: tp += 1
        else: fp += 1
    for gold in gold_set:
        if gold not in pred_set: fn += 1
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Strict vs. lenient matching criteria differ significantly for nested/overlapped concepts; using the wrong script yields misleading F1 scores.
• Metrics must be aggregated micro-averaged across all concept and relation categories, not macro-averaged or per-category.
• Official n2c2 evaluation scripts must be used; custom implementations often diverge on boundary handling.

Evidence (verbatim from paper)

We reported the strict and lenient micro-averaged precision, recall, and F1-score aggregated from all concept and relation categories. The official evaluation scripts provided by the 2018 n2c2 challenges were used to calculate the scores.

Citation

@misc{peng2023clinicalconcept,
  title={Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension},
  author={Peng et al. (2023)},
  year={2023},
  note={arXiv:2303.08262}
}

• arXiv: 2303.08262