Entity Linking Eval

entity-linking-eval

A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems — Bast et al. (2023) (arXiv:2305.14937, 2023)

What this evaluates

Evaluates end-to-end entity linking systems on their ability to detect entity mentions and correctly disambiguate them to knowledge base entities. It specifically probes for systemic benchmark biases, such as overreliance on named entities, ambiguous disambiguation choices, and underrepresented entity types, by introducing fairer evaluation protocols.

Datasets

• Existing and new EL benchmarks — total ?; splits: test (-1); repo https://github.com/ad-freiburg/fair-entity-linking-benchmarks

Metrics

• Micro F1 (primary) — range: [0, 1]
  • Micro-averaged F1 score computed over all mention-level predictions across the dataset. Global precision and recall are aggregated before computing F1.
• Disambiguation Accuracy — range: [0, 1]
  • The number of correctly linked entities divided by the number of correctly detected entity mentions. Ignores detection errors.
• Disambiguation Error Rate — range: [0, 1]
  • One minus the disambiguation accuracy.

Input / output format

Input: Document text with annotated mention spans and candidate entity lists.

Output: Predicted entity ID for each mention span, or NIL if no entity matches.

Scoring recipe

tp = sum(1 for p, g in zip(preds, gold) if p == g and p != 'NIL')
fp = sum(1 for p, g in zip(preds, gold) if p != g and p != 'NIL')
fn = sum(1 for p, g in zip(preds, gold) if p == 'NIL' and g != 'NIL')
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
detected = tp + fn
linked = tp
disamb_acc = linked / detected if detected > 0 else 0

Common pitfalls

• Micro-averaging can mask poor performance on rare or ambiguous entity types.
• Disambiguation accuracy is computed only on correctly detected mentions, ignoring detection errors.
• Fine-grained error categories (e.g., 'Lowercased', 'Ground truth NIL') require strict adherence to the ELEVANT tool's definitions to avoid misclassification.

Evidence (verbatim from paper)

We report micro precision, recall and F1 scores, both for the overall EL task and for the ER subtask. ... The disambiguation accuracy is defined as the correctly linked entities divided by the correctly detected entity mentions.

Citation

@misc{bast2023fair,
  title={A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems},
  author={Bast et al. (2023)},
  year={2023},
  note={arXiv:2305.14937}
}

• arXiv: 2305.14937