Conve Kg Link Prediction Eval

conve-kg-link-prediction-eval

Convolutional 2D Knowledge Graph Embeddings — Dettmers et al. (2017) (arXiv:1707.01476, 2017)

What this evaluates

Evaluates knowledge graph link prediction by measuring a model's ability to infer missing entities (head or tail) from subject-relation triples. It probes the expressiveness of 2D convolutional embeddings and tests robustness against test-set leakage via inverse relations.

Datasets

• WN18 — total 151442; splits: train (-1), val (-1), test (-1)
• FB15k — total ?; splits: train (-1), val (-1), test (-1)
• YAGO3-10 — total ?; splits: train (-1), val (-1), test (-1)
• Countries — total ?; splits: train (-1), val (-1), test (-1)
• FB15k-237 — total ?; splits: train (-1), val (-1), test (-1)
• WN18RR — total 93003; splits: train (-1), val (-1), test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal ranks of the correct entity across all test triples. Calculated as (1/N) * Σ (1/rank_i).
• Hits@1 — range: [0, 1]
  • Fraction of test triples where the correct entity appears at rank 1 in the predicted ranking.
• Hits@3 — range: [0, 1]
  • Fraction of test triples where the correct entity appears in the top 3 predicted ranks.
• Hits@10 — range: [0, 1]
  • Fraction of test triples where the correct entity appears in the top 10 predicted ranks.
• AUC-PR — range: [0, 1]
  • Area Under the Precision-Recall Curve, used specifically for the Countries dataset due to its high variance and multi-task nature.

Input / output format

Input: Triple (subject, relation, object) with one entity masked (head or tail prediction), evaluated against a candidate set of all entities in the knowledge graph.

Output: Ranked list of candidate entities sorted by predicted score, or top-k predictions.

Scoring recipe

ranks = []
for triple in test_set:
    scores = model.predict(triple)
    rank = rank_of_correct_entity(scores)
    ranks.append(rank)
mrr = sum(1.0 / r for r in ranks) / len(ranks)
hits_at_k = sum(1 for r in ranks if r <= k) / len(ranks)

Common pitfalls

• Test set leakage via inverse relations causes inflated scores on original WN18 and FB15k; must use filtered versions (WN18RR, FB15k-237).
• Countries dataset exhibits high variance; results must be averaged over 10 runs with 95% confidence intervals.
• Early stopping is based on validation MRR/AUC-PR, not test set performance.

Evidence (verbatim from paper)

We selected the hyperparameters of our ConvE model via grid search according to the mean reciprocal rank (MRR) on the validation set. ... We use early stopping according to the mean reciprocal rank (WN18, FB15k, YAGO3-10) and AUC-PR (Countries) statistics on the validation set, which we evaluate every three epochs. Unlike the other datasets, for Countries the results have a high variance, as such we average 10 runs and produce 95% confidence intervals.

Citation

@misc{dettmers2017conve,
  title={Convolutional 2D Knowledge Graph Embeddings},
  author={Dettmers et al. (2017)},
  year={2017},
  note={arXiv:1707.01476}
}

• arXiv: 1707.01476