Entities Of The Union Eval

entities-of-the-union-eval

Contrastive Entity Coreference and Disambiguation for Historical Texts — Arora et al. (2024) (arXiv:2406.15576, 2024)

What this evaluates

Evaluates the ability of models to disambiguate named entity mentions in historical and modern newswire texts, and to cluster coreferent mentions across documents. It specifically probes handling of out-of-knowledgebase individuals common in historical contexts.

Datasets

• Entities of the Union — total ?; splits: test (-1)
• MSNBC — total ?; splits: test (-1)
• ACE2004 — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly disambiguated entity mentions out of the total number of mentions. Calculated as (correct predictions / total mentions) * 100.
• adjusted rand index (ARI) — range: [0, 1]
  • A clustering similarity metric that measures the agreement between predicted coreference clusters and ground truth labels, adjusted for chance.

Input / output format

Input: Text snippets containing named entity mentions from historical or modern newswire articles, along with surrounding context.

Output: Disambiguated entity identifier (e.g., Wikipedia ID) or a classification indicating the mention is out-of-knowledgebase. For coreference, cluster assignments for mentions across documents.

Scoring recipe

def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_ari(pred_clusters, gold_clusters):
    from sklearn.metrics import adjusted_rand_score
    return adjusted_rand_score(gold_clusters, pred_clusters)

Common pitfalls

• Baseline models like LUKE are restricted to the top 50K Wikipedia entities, making them unsuitable for evaluating out-of-knowledgebase historical mentions.
• Code for many existing disambiguation models is unavailable or unmaintained, requiring careful reproduction of their implementations.
• Coreference resolution is treated as a preprocessing step rather than a primary evaluated task, so direct coreference benchmarks are not reported.

Evidence (verbatim from paper)

LinkNewsWikipedia correctly retrieves or classifies as out-of-knowledgebase 78% of individual mentions, whereas LinkWikipedia has an accuracy of 74%. The next best alternative is ReFinED, which correctly disambiguates around 65% of mentions. When only considering entities in Wikipedia, LinkNewsWikipedia correctly disambiguates 89% of entities, LinkWikipedia correctly disambiguates 85% of entities, and the next best alternative is GENRE, with an accuracy of 81%.

Citation

@misc{arora2024contrastive,
  title={Contrastive Entity Coreference and Disambiguation for Historical Texts},
  author={Arora et al. (2024)},
  year={2024},
  note={arXiv:2406.15576}
}

• arXiv: 2406.15576