Entity Canonicalization Eval

entity-canonicalization-eval

Open Knowledge Graphs Canonicalization using Variational Autoencoders — Dash et al. (2020) (arXiv:2012.04780, 2020)

What this evaluates

Evaluates a model's ability to cluster entity mentions (noun phrases) into canonical forms within open knowledge graphs. It probes unsupervised representation learning and clustering capabilities without relying on manually annotated ground truth for training.

Datasets

• Base — total 290; splits: test (-1), val (-1); repo https://github.com/mallabiisc/cesi/
• Ambiguous — total 717; splits: test (-1), val (-1); repo https://github.com/mallabiisc/cesi/
• ReVerb45K — total 15500; splits: test (-1), val (-1); repo https://github.com/mallabiisc/cesi/
• CanonicNell — total 8700; splits: test (-1), val (-1); repo https://github.com/IBM/Open-KG-canonicalization

Metrics

• Macro F1 (primary) — range: [0, 1]
  • Averages the F1 score computed independently for each cluster, giving equal weight to clusters regardless of their size.
• Micro F1 — range: [0, 1]
  • Aggregates true positives, false positives, and false negatives across all clusters to compute a global F1 score.
• Pair F1 — range: [0, 1]
  • Evaluates clustering by considering all pairs of items. Counts true positives if pairs are in the same cluster in both predictions and gold, and false positives/negatives otherwise.

Input / output format

Input: Noun phrases (entity mentions) from KG triples, optionally augmented with side information (e.g., IDF token overlap, pretrained embeddings, structural graph features).

Output: A cluster identifier (canonical label) assigned to each noun phrase.

Scoring recipe

def compute_f1_metrics(predictions, gold):
    # predictions, gold: lists of cluster IDs per noun phrase
    macro_f1 = mean(f1_per_cluster(p, g) for p, g in zip(predictions, gold))
    micro_f1 = global_precision_recall_f1(predictions, gold)
    pair_f1 = pair_f1_score(predictions, gold)
    mean_f1 = (macro_f1 + micro_f1 + pair_f1) / 3
    return {'macro_f1': macro_f1, 'micro_f1': micro_f1, 'pair_f1': pair_f1, 'mean_f1': mean_f1}

Common pitfalls

• The task is strictly unsupervised; no training data is used, so hyperparameters are tuned solely on the validation set.
• Initial evaluations only considered head entity mentions; tail mentions were added later, causing score discrepancies across benchmarks.
• Pair F1 is highly sensitive to cluster size imbalance and often reports significantly lower values than Macro/Micro F1 on the same dataset.

Evidence (verbatim from paper)

Following Galárraga et al. (2014a), we use the macro, micro, and pair F1 scores for evaluations. The task is unsupervised in nature, hence we do not possess any training data. For CanonicNell, we did a random 80:20 split of the triples into validation and test folds.

Citation

@misc{dash2020openkgcanonicalization,
  title={Open Knowledge Graphs Canonicalization using Variational Autoencoders},
  author={Dash et al. (2020)},
  year={2020},
  note={arXiv:2012.04780}
}

• arXiv: 2012.04780