Kb Completion Eval

kb-completion-eval

Learn to Explain Efficiently via Neural Logic Inductive Learning — Yang et al. (2019) (arXiv:1910.02481, 2019)

What this evaluates

Evaluates a model's ability to learn first-order logic rules for knowledge base completion and object classification. It probes rule generation efficiency, scalability to longer rules, and few-shot generalization on relational data.

Datasets

• Even-and-Successor (ES) — total ?; splits: train (-1), valid (-1), test (-1)
• FB15K-237 — total ?; splits: train (-1), valid (-1), test (-1)
• WN18 — total ?; splits: train (-1), valid (-1), test (-1)
• Visual Genome (via GQA) — total ?; splits: train (-1), test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank for the correct answer across all queries.
• Hits@10 — range: [0, 1]
  • Fraction of queries where the correct answer is ranked in the top 10 predictions.
• R@1 — range: [0, 1]
  • Accuracy of predicting the correct object class label at rank 1.
• R@5 — range: [0, 1]
  • Accuracy of predicting the correct object class label within the top 5 predictions.

Input / output format

Input: Relational knowledge base triples (subject, relation, object) or scene-graphs; query triplets for KB completion; object-relation pairs for classification.

Output: Probability score for a fact triplet being present in the KB; predicted object class label.

Scoring recipe

def compute_metrics(predictions, gold_indices):
    ranks = []
    for pred, gold in zip(predictions, gold_indices):
        sorted_indices = np.argsort(-pred)
        rank = np.where(sorted_indices == gold)[0][0] + 1
        ranks.append(rank)
    mrr = np.mean(1.0 / ranks)
    hits10 = np.mean([r <= 10 for r in ranks])
    r1 = np.mean([r == 1 for r in ranks])
    r5 = np.mean([r <= 5 for r in ranks])
    return mrr, hits10, r1, r5

Common pitfalls

• The Even-and-Successor benchmark is noise-free, so the paper only reports wall-clock time for it rather than accuracy scores.
• The Visual Genome dataset is highly noisy; the authors use a pre-processed GQA version and filter predicates with fewer than 1500 occurrences, which alters the original data distribution.

Evidence (verbatim from paper)

We use Mean Reciprocal Ranks (MRR) and Hits@10 for evaluation metrics (see AppendixC for details). Quantitatively, we evaluate the learned rules on predicting the object class labels on a held-out set in terms of their R@1 and R@5.

Citation

@misc{yang2019learn,
  title={Learn to Explain Efficiently via Neural Logic Inductive Learning},
  author={Yang et al. (2019)},
  year={2019},
  note={arXiv:1910.02481}
}

• arXiv: 1910.02481