Medical Entity Linking Eval

medical-entity-linking-eval

Improving Broad-Coverage Medical Entity Linking with Semantic Type Prediction and Large-Scale Datasets — Vashishth et al. (2020) (arXiv:2005.00460, 2020)

What this evaluates

Evaluates a model's ability to predict semantic types for biomedical mentions and to link those mentions to standardized medical concepts. It probes how well type-based candidate filtering improves broad-coverage medical information extraction pipelines.

Datasets

• NCBI Disease Corpus — total ?; splits: test (-1)
• Bio CDR — total ?; splits: test (-1)
• ShARE — total ?; splits: test (-1)
• MedMentions — total ?; splits: test (-1)
• WIKIMED — total ?; splits: train (-1)
• PUBMEDDS — total ?; splits: train (-1)

Metrics

• AUC (Area Under the Precision-Recall curve) (primary) — range: [0, 1]
  • Area under the precision-recall curve computed across classification thresholds for semantic type prediction.
• Exact Mention_id_MATCH F1 — range: [0, 1]
  • F1-score calculated on exact matches of mention spans and their linked concept IDs.
• Partial Mention_id_MATCH F1 — range: [0, 1]
  • F1-score calculated on partial matches of mention spans and their linked concept IDs, used to isolate entity linking performance from mention detection errors.

Input / output format

Input: Biomedical text snippets containing medical concept mentions, optionally with candidate concept lists for entity linking.

Output: Predicted semantic type labels (for type prediction) or ranked candidate concepts with CUIs (for entity linking).

Scoring recipe

def compute_f1(preds, golds, match_type='exact'):
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        if match_type == 'exact' and p == g: tp += 1
        elif match_type == 'partial' and spans_overlap(p, g): tp += 1
        else: fp += 1
    fn = len(golds) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_auc(pr_scores, labels):
    return auc(pr_curve(labels, pr_scores))

Common pitfalls

• Evaluating predictions for semantic types excluded from a dataset's annotation scheme (e.g., non-disease types on NCBI) leads to invalid scores; predictions must be filtered to the dataset's defined types.
• Failing to separate mention detection errors from entity linking errors; the paper isolates linking by restricting evaluation to predicted mentions that overlap with gold annotations.
• Assuming fine-grained typing is strictly necessary; the paper shows coarse-grained oracle types yield negligible performance difference compared to fine-grained ones.

Evidence (verbatim from paper)

Table 6 reports the results for the Exact Mention_id_MATCH and Partial Mention_id_MATCH metrics, as described in Section 5.4. ... we report the area under the precision-recall curve as our evaluation metric.

Citation

@misc{vashishth2020improving,
  title={Improving Broad-Coverage Medical Entity Linking with Semantic Type Prediction and Large-Scale Datasets},
  author={Vashishth et al. (2020)},
  year={2020},
  note={arXiv:2005.00460}
}

• arXiv: 2005.00460