Wiki Alumni Eval

wiki-alumni-eval

Node Classification Meets Link Prediction on Knowledge Graphs — Abboud et al. (2021) (arXiv:2106.07297, 2021)

What this evaluates

Evaluates knowledge graph models on node and entity classification under edge incompleteness, and link prediction with varying feature/class configurations. It probes how well models leverage relational structure, node features, and joint link prediction capacity to handle missing edges and predict missing links.

Datasets

• WikiAlumni — total ?; splits: 100% (-1), 90% (-1), 80% (-1)

Metrics

• validation accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted node or entity classes out of the total instances in the validation set.
• mean rank (MR) — range: [1, N]
  • Average rank of the true tail entity across all test triples, ranked by model score.
• mean reciprocal rank (MRR) — range: [0, 1]
  • Average of the reciprocal of the rank of the true tail entity for each test triple.
• Hits@10 (H@10) — range: [0, 1]
  • Fraction of test triples where the true tail entity appears in the top 10 ranked predictions.

Input / output format

Input: Knowledge graph instances containing entity node features, edge structure (relations), and optional class labels. Models are evaluated in configurations with or without node features and node classes.

Output: For classification: a predicted class label. For link prediction: a continuous score for each candidate entity used to generate a ranked list.

Scoring recipe

def compute_metrics(predictions, golds):
    # Classification
    accuracy = sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    
    # Link Prediction
    ranks = [rank_of_true_entity(entity_scores) for entity_scores in predictions]
    mr = sum(ranks) / len(ranks)
    mrr = sum(1.0 / r for r in ranks) / len(ranks)
    hits10 = sum(1 if r <= 10 else 0 for r in ranks) / len(ranks)
    return accuracy, mr, mrr, hits10

Common pitfalls

• Confusing 'node classification' (uses node features + edges) with 'entity classification' (uses only edges/embeddings, explicitly discards features).
• Assuming GNN baselines and MLP-X models are directly comparable without noting MLP-X's joint link prediction capacity versus GNNs' distinct link prediction setup.
• Misinterpreting BoxE scoring direction: lower scores indicate higher confidence, contrary to most embedding models.

Evidence (verbatim from paper)

Finally, we evaluate using standard metrics, namely mean rank (MR), mean reciprocal rank (MRR), and Hits@10 (H@10) [[3]].

Citation

@misc{abboud2021nodeclassification,
  title={Node Classification Meets Link Prediction on Knowledge Graphs},
  author={Abboud et al. (2021)},
  year={2021},
  note={arXiv:2106.07297}
}

• arXiv: 2106.07297