Llmke Wikidata Eval

llmke-wikidata-eval

Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata — Zhang et al. (2023) (arXiv:2309.08491, 2023)

What this evaluates

Evaluates LLMs' ability to predict object entities given subject-relation pairs in Wikidata, testing knowledge retrieval, entity disambiguation, and domain-specific reasoning across 21 relations spanning 7 domains.

Datasets

• ISWC 2023 LM-KBC Challenge dataset — total ?; splits: train (1940), val (1940), test (1940); repo https://github.com/bohuizhang/LLMKE

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (P * R) / (P + R). Computed per relation and averaged macroscopically across the 21 relations.

Input / output format

Input: Subject entity, relation type, and optionally few-shot examples or retrieved context (e.g., Wikipedia introduction/infobox, IMDb data) formatted as prompts for question answering or triple completion.

Output: Predicted object entity string, which is then mapped to a Wikidata QID via disambiguation methods (baseline, improved keyword/LM/case-based).

Scoring recipe

def compute_macro_f1(predictions, golds):
    relation_f1s = []
    for rel in relations:
        preds_rel = [p for p, g in zip(predictions, golds) if g['rel'] == rel]
        golds_rel = [g for p, g in zip(predictions, golds) if g['rel'] == rel]
        tp = sum(1 for p, g in zip(preds_rel, golds_rel) if p in g)
        fp = sum(1 for p, g in zip(preds_rel, golds_rel) if p not in g)
        fn = sum(1 for p, g in zip(preds_rel, golds_rel) if not any(gi in p for gi in g))
        p = tp / (tp + fp) if (tp + fp) > 0 else 0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0
        relation_f1s.append(2 * p * r / (p + r) if (p + r) > 0 else 0)
    return sum(relation_f1s) / len(relation_f1s)

Common pitfalls

• Disambiguation errors: LLMs may predict correct entity strings but assign wrong QIDs, or use aliases absent from Wikidata's label/alias list.
• Context mismatch: Retrieval-augmented context from Wikipedia may not align with Wikidata's entity representations, sometimes hurting performance.
• Multi-answer handling: Relations can have 0 to 20 valid objects; F1 calculation must account for variable answer sets per instance.

Evidence (verbatim from paper)

It has 1,940 statements for each train, validation, and test sets. The results reported are based on the test set. In the dataset, the minimum and maximum number of object-entities for each relation is different, ranging from 0 to 20. The best F1-scores among the three settings and two disambiguation methods of the models are highlighted.

Citation

@misc{zhang2023llmke,
  title={Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2309.08491}
}

• arXiv: 2309.08491