Clinical Ner Eval

clinical-ner-eval

Named Clinical Entity Recognition Benchmark — Abdul et al. (2024) (arXiv:2410.05046, 2024)

What this evaluates

Evaluates language models' ability to identify and classify standardized medical entities (e.g., diseases, drugs, procedures, genes) in unstructured clinical text. It probes sequence labeling performance under strict terminology standardization (OMOP CDM) to ensure interoperability across diverse healthcare datasets.

Datasets

• NCBI Disease corpus — total 100; splits: test (-1)
• CHIA — total 194; splits: test (-1)
• BC5CDR — total 500; splits: test (-1)
• BIORED — total 100; splits: test (-1)

Metrics

• Macro Average F1-score (token-based) (primary) — range: [0, 1]
  • Precision and recall are calculated per entity type, then averaged without weighting. Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2*(P*R)/(P+R).
• Partial Match F1-score (span-based) — range: [0, 1]
  • Evaluates at the entity level. A predicted span is a true positive if it overlaps with the true span's boundary and exactly matches the label. Precision, Recall, and F1 are computed using these span-level counts.

Input / output format

Input: A sequence of tokens X=(x_1, ..., x_n) extracted from unstructured clinical text (e.g., patient notes, PubMed abstracts, clinical trial eligibility criteria).

Output: A sequence of BIO-style labels Y=(y_1, ..., y_n) where each y_i ∈ {B-, I-, O} combined with entity types (e.g., B-DIS, I-DRUG, O).

Scoring recipe

def compute_macro_token_f1(predictions, golds, entity_types):
    tp, fp, fn = {e:0 for e in entity_types}, {e:0 for e in entity_types}, {e:0 for e in entity_types}
    for pred_labels, gold_labels in zip(predictions, golds):
        for p, g in zip(pred_labels, gold_labels):
            if p == g: tp[g] += 1
            elif p != 'O' and g == 'O': fp[p] += 1
            elif p == 'O' and g != 'O': fn[g] += 1
    f1s = []
    for etype in entity_types:
        prec = tp[etype] / (tp[etype] + fp[etype]) if (tp[etype] + fp[etype]) > 0 else 0
        rec = tp[etype] / (tp[etype] + fn[etype]) if (tp[etype] + fn[etype]) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec))
    return sum(f1s) / len(f1s)

Common pitfalls

• Token-level metrics can misrepresent performance for multi-token entities, as they penalize boundary mismatches heavily.
• Exact span matching is often too strict for clinical text; partial match (allowing boundary overlap with correct label) is preferred for real-world applicability.
• Datasets use heterogeneous entity typologies; failing to map them to the standardized OMOP CDM framework will break cross-dataset comparability.

Evidence (verbatim from paper)

Given an input sequence of tokens $X=(x_{1},x_{2},\ldots,x_{n})$, where each $x_{i}$ represents a token (a word or sub-word) in clinical text, the goal is to assign a corresponding sequence of labels $Y=(y_{1},y_{2},\ldots,y_{n})$, where each $y_{i}$ belongs to a predefined set of clinical entity types $E\cup{O}$... For our evaluation framework we consider the Macro Average token-based metrics and the Partial Match for our span-based metrics.

Citation

@misc{abdul2024namedclinical,
  title={Named Clinical Entity Recognition Benchmark},
  author={Abdul et al. (2024)},
  year={2024},
  note={arXiv:2410.05046}
}

• arXiv: 2410.05046