LLM Kg Bench 3.0 Eval

llm-kg-bench-3.0-eval

LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMs — Meyer et al. (2025) (arXiv:2505.13098, 2025)

What this evaluates

Evaluates Large Language Models' proficiency in Knowledge Graph Engineering tasks, specifically focusing on RDF syntax repair, SPARQL query generation and semantics, and data serialization format handling across multiple graph structures.

Datasets

• LLM-KG-Bench 3.0 — total ?; splits: RdfSyntaxFixList (-1), RdfConnectionExplainStatic (-1), RdfFriendCount (-1), SparqlSyntaxFixingList (-1), Sparql2AnswerList (-1), Text2SparqlList (-1); repo https://github.com/AKSW/LLM-KG-Bench

Metrics

• capability compass (primary) — range: [0, 1]
  • Aggregated performance score across five configured dimensions (Brevity, RDF Syntax, RDF Analytics, SPARQL Semantics, SPARQL Syntax) derived from task-level correctness. Exact per-task formulas are not specified but rely on syntactic validity and semantic correctness of generated RDF/SPARQL.

Input / output format

Input: Task-specific prompts containing instructions, graph data in various serializations (Turtle, JSON-LD, N-Triples, RDF/XML), and context for RDF repair, connection explanation, friend counting, or SPARQL query generation.

Output: Generated RDF code, SPARQL queries, or textual answers corresponding to the task instructions.

Scoring recipe

def compute_capability_compass_score(predictions, gold, task_type):
    if task_type in ['RdfSyntaxFixList', 'SparqlSyntaxFixingList']:
        correct = is_syntactically_valid(predictions) and matches_gold_semantics(predictions, gold)
    elif task_type in ['Sparql2AnswerList', 'Text2SparqlList']:
        correct = exact_match_or_semantically_equivalent(predictions, gold)
    else:
        correct = exact_match(predictions, gold)
    return 1.0 if correct else 0.0
# Aggregate correct predictions per task into capability compass dimensions

Common pitfalls

• Iteration counts differ between proprietary (20) and open (50) LLMs, which may bias statistical comparisons and confidence intervals.
• Format preference evaluation (TTL vs JSON-LD) relies on two-sided t-tests with varying confidence thresholds (95% vs 99%), making cross-model comparisons sensitive to statistical power.
• Context window limits excluded capable models (e.g., solar-pro-preview-instruct), potentially skewing open-model leaderboards and capability assessments.

Evidence (verbatim from paper)

In LLM-KG-Bench framework Version 3, we added the capability to aggregate results for each model evaluated and create capability compass plots. We used an exemplary configuration to create the ones shown in [fig. 4]. These plots can be used to give a summary of a model or create model cards.

Citation

@misc{meyer2025llmkgbench,
  title={LLM-KG-Bench 3.0: A Compass for SemanticTechnology Capabilities in the Ocean of LLMs},
  author={Meyer et al. (2025)},
  year={2025},
  note={arXiv:2505.13098}
}

• arXiv: 2505.13098