Inductive Link Prediction Eval

inductive-link-prediction-eval

Inductive Link Prediction in Knowledge Graphs using Path-based Neural Networks — Zhang et al. (2023) (arXiv:2312.10293, 2023)

What this evaluates

Evaluates a model's ability to predict missing links in knowledge graphs using only topological path information, without relying on entity embeddings. It tests inductive generalization by training on one graph and testing on a disjoint graph with unseen entities.

Datasets

• WN18RR, FB15K-237, NELL-995 (inductive versions v1-v4) — total ?; splits: train (-1), test (-1)

Metrics

• AUC-PR — range: percent
  • Replace the source or target entity of each test triple with a random entity to form negative triples. Score the positive test triples and an equal number of negative triples, then compute the area under the precision-recall curve.
• Hits@10 — range: percent
  • Rank each test triple against corrupted candidates. In entity-corrupted ranking, candidates are formed by replacing the source or target entity with a random entity (50 negatives per test triple). The metric is the percentage of true triples ranked in the top-10 positions.
• Hits@1 (primary) — range: percent
  • Rank each test triple against corrupted candidates where the relation is replaced by other relations in the graph. The metric is the percentage of true triples ranked in the top-1 position.
• Hits@3 — range: percent
  • Same as Hits@1 but measures the percentage of true triples ranked in the top-3 positions under relation-corrupted ranking.

Input / output format

Input: Path sequences connecting source and target entities (connection-based) or out-reaching paths from source/target entities (subgraph-based), optionally paired with relation embeddings.

Output: A scalar similarity score (inner product) for each candidate triple, used to rank candidates.

Scoring recipe

def evaluate(dataset, model):
    hits_at_10 = []
    auc_pr_pairs = []
    for triple in dataset.test_triples:
        # Entity-corrupted negatives (50 fixed)
        negatives = generate_entity_corrupted(triple, k=50)
        candidates = [triple] + negatives
        scores = [model.score(c) for c in candidates]
        true_score = scores[0]
        rank = sum(1 for s in scores[1:] if s > true_score) + 1
        hits_at_10.append(1 if rank <= 10 else 0)
        # AUC-PR uses 1:1 ratio
        neg_1 = generate_entity_corrupted(triple, k=1)[0]
        auc_pr_pairs.append((true_score, 1))
        auc_pr_pairs.append((model.score(neg_1), 0))
    return {
        'Hits@10': sum(hits_at_10) / len(hits_at_10) * 100,
        'AUC-PR': compute_auc(auc_pr_pairs) * 100
    }

Common pitfalls

• Entity-corrupted and relation-corrupted ranking use fundamentally different negative sampling strategies (fixed 50 negatives vs. variable negatives based on relation count), making Hits@k values incomparable across settings.
• Inductive splits enforce strictly disjoint entity sets between training and inference graphs, so models must rely purely on topological paths rather than pre-trained entity embeddings.
• AUC-PR evaluation uses a strict 1:1 positive-to-negative ratio, which differs from standard open-world link prediction benchmarks and can inflate or deflate scores depending on dataset density.

Evidence (verbatim from paper)

We apply both classification metric and ranking metric to evaluate the performance of our model. For classification metric, we use the area under the precision-recall curve (AUC-PR) following GraIL. That is, we replace the source or target entity of each test triple with a random entity to form a negative triple. Then, we score the positive test triples with an equal number of negative triples to calculate AUC-PR. For the ranking metric, however, there seems to be two different settings. The first setting is purposed in GraIL... Accordingly, Hits@10 (the rate of true test triples ranked top-10 in all performed rankings) is calculated with respect to all test triples. We refer to this setting as entity-corrupted ranking.

Citation

@misc{zhang2023inductive,
  title={Inductive Link Prediction in Knowledge Graphs using Path-based Neural Networks},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2312.10293}
}

• arXiv: 2312.10293