Wikidata Ned Eval

wikidata-ned-eval

Named Entity Disambiguation using Deep Learning on Graphs — Alberto Cetoli et al. (arXiv:1810.09164, 2018)

What this evaluates

Evaluates a model's ability to disambiguate named entities in text by matching them to correct Wikidata entries using graph-based representations. It probes how well different neural architectures leverage graph triplet information versus full graph topology for entity resolution.

Datasets

• Wikidata-Disamb — total ?; splits: dev (-1), test (-1); repo https://github.com/contextscout/ned-graphs

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). Reported as a percentage.

Input / output format

Input: Input text containing a named entity mention, accompanied by a set of graph triplets representing the candidate entity's Wikidata graph.

Output: Disambiguated entity identifier (Wikidata ID) or classification label indicating the correct entity from the candidate set.

Scoring recipe

def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)
# F1 is computed per model on dev/test splits and reported as percentage

Common pitfalls

• GCN-based models underperform because they overemphasize graph topology/shape rather than triplet content, which is the key signal in this dataset.
• Statistical significance is not rigorously tested; the authors estimate error at 0.6% based on only two runs per model.
• The dataset consists of simple tree-like graphs, so models relying on complex topological features may not generalize to richer graph structures.

Evidence (verbatim from paper)

The simple vector distance baseline is seen here performing narrowly better than random chance, with F1 = 55.4% on the test set. ... The second best results of the paper is given by the RNN of triplets model, with F1 = 91.1% on the test set. ... Table 3: Results of our architectures expressed as a percentage (best results in bold).

Citation

@misc{cetoli2018nedgraphs,
  title={Named Entity Disambiguation using Deep Learning on Graphs},
  author={Alberto Cetoli et al.},
  year={2018},
  note={arXiv:1810.09164}
}

• arXiv: 1810.09164