Mokb6 Eval

mokb6-eval

mOKB6: A Multilingual Open Knowledge Base Completion Benchmark — Mittal et al. (2022) (arXiv:2211.06959, 2022)

What this evaluates

This benchmark evaluates multilingual knowledge graph embedding models on the task of completing missing facts across six languages. It specifically probes the model's ability to leverage cross-lingual information flow, benefit from translated training triples, and retain facts when queried in different scripts.

Datasets

• mOKB6 — total ?; splits: test (-1); repo https://github.com/dair-iitd/mokb6.git

Metrics

• H@1 — range: percent
  • Hits@1 measures the percentage of queries where the correct entity is ranked exactly first among all candidates.
• H@10 (primary) — range: percent
  • Hits@10 measures the percentage of queries where the correct entity is ranked within the top 10 predictions.
• MRR — range: percent
  • Mean Reciprocal Rank computes the average of the reciprocal ranks (1/rank) of the correct entity across all queries.

Input / output format

Input: A query triple (head entity, relation) with a candidate tail entity to be ranked among a set of negative samples.

Output: A ranked list of candidate tail entities (or their scores) ordered by predicted likelihood.

Scoring recipe

def compute_metrics(ranks, n_queries):
    h10 = sum(1 for r in ranks if r <= 10) / n_queries * 100
    mrr = sum(1.0 / r for r in ranks) / n_queries * 100
    h1 = sum(1 for r in ranks if r == 1) / n_queries * 100
    return h1, h10, mrr

Common pitfalls

• The test sets across the six languages contain identical underlying facts, so evaluating cross-lingual memorization requires training on one language's test set and testing on others, rather than using standard disjoint train/test splits.
• Performance heavily depends on script similarity (e.g., Latin vs. Devanagari/Han); models trained on Latin-script languages transfer poorly to non-Latin scripts, which can skew cross-lingual transfer conclusions if not accounted for.
• The 'Union' and 'Union+Trans' setups mix facts from all six languages during training, which can cause language interference or bias toward high-resource languages like English if not properly normalized.

Evidence (verbatim from paper)

For evaluation, we use three metrics — hits at rank 1 (H@1), hits at rank 10 (H@10), and mean reciprocal rank (MRR).

Citation

@misc{mittal2022mokb6,
  title={mOKB6: A Multilingual Open Knowledge Base Completion Benchmark},
  author={Mittal et al. (2022)},
  year={2022},
  note={arXiv:2211.06959}
}

• arXiv: 2211.06959