Irt2 Eval

irt2-eval

IRT2: Inductive Linking and Ranking in Knowledge Graphs of Varying Scale — Hamann et al. (2023) (arXiv:2301.00716, 2023)

What this evaluates

Evaluates neural and baseline models on inductive link prediction and ranking tasks across knowledge graphs of varying scales. It probes the models' ability to map textual entity mentions to graph vertices and rank candidate entities based on combined textual and structural signals, particularly under data scarcity conditions.

Datasets

• IRT2 — total ?; splits: tiny (-1), small (-1), medium (-1), large (-1); repo https://github.com/lavis-nlp/irt2

Metrics

• hits@10 — range: [0, 1]
  • Fraction of test triples where the true entity is ranked within the top 10 predictions after target filtering.
• hits@100 — range: [0, 1]
  • Fraction of test triples where the true entity is ranked within the top 100 predictions after target filtering.
• MRR (primary) — range: [0, 1]
  • Mean reciprocal rank of the true entity across all test triples. Calculated as the average of 1/rank for each query, where rank is the position of the true entity in the filtered prediction list.

Input / output format

Input: Textual entity mentions (sentences) paired with a knowledge graph structure. For linking, a text context and a target graph; for ranking, a text context and a query relation.

Output: A ranked list of candidate entities (vertices) or a binary decision indicating the correct graph vertex for the mention.

Scoring recipe

def compute_metrics(predictions, gold, k=10):
    # Apply target filtering: keep only the single true positive of interest
    filtered = [x for x in predictions if x == gold] + \
               [x for x in predictions if x != gold and x not in other_true_positives]
    rank = filtered.index(gold) + 1 if gold in filtered else len(filtered) + 1
    hits_k = 1.0 if rank <= k else 0.0
    mrr = 1.0 / rank
    return hits_k, mrr
# Aggregate micro-averaged over all test triples

Common pitfalls

• Target filtering must be applied to remove other true positives from the ranked list to avoid artificially worsening results for rankings with many true positives.
• The ranking task relies heavily on head prediction scores, which are often lower quality than tail predictions, leading to poor practical ranking performance despite neural models beating BM25.
• Models struggle to incorporate textual clues and rely heavily on structural embeddings, especially in larger datasets where the bag-of-words baseline can outperform them.

Evidence (verbatim from paper)

We measure hits@k and the mean reciprocal rank (MRR) for each test triple (i.e. micro-averaged). Following common practice, we apply target filtering [Sh19] to the scored predictions (mentions for ranking and vertices for ranking). Target filtering is a method where, for a given ranking, only a single true positive of interest is inspected, while all other true positives are removed from the list. This helps to not artificially worsen the result for rankings with many true positives.

Citation

@misc{hamann2023irt2,
  title={IRT2: Inductive Linking and Ranking in Knowledge Graphs of Varying Scale},
  author={Hamann et al. (2023)},
  year={2023},
  note={arXiv:2301.00716}
}

• arXiv: 2301.00716