Kgqa4mat Eval

kgqa4mat-eval

Knowledge Graph Question Answering for Materials Science (KGQA4MAT): Developing Natural Language Interface for Metal-Organic Frameworks Knowledge Graph (MOF-KG) Using LLM — Yuan An et al. (2023) (arXiv:2309.11361, 2023)

What this evaluates

Evaluates a model's ability to translate natural language questions into correct graph database queries (Cypher or SPARQL) for knowledge graph question answering. It probes the model's capacity for logical reasoning, schema understanding, and formal language generation in both a domain-specific materials science setting and a general-domain multilingual benchmark.

Datasets

• KGQA4MAT — total 644; splits: test (644); repo https://github.com/kgqa4mat/KGQA4MAT
• QALD-9 — total 558; splits: train (408), test (150)

Metrics

• F1-score (primary) — range: [0, 1]
  • F1 = 2 * (1/precision + 1/recall)^-1. Precision and recall are computed by executing the translated query and comparing its result set to the gold query's result set. Due to generating exactly one query per question, precision equals recall equals F1.

Input / output format

Input: Natural language question, optionally augmented with knowledge graph ontology definitions, a 1-shot question-query pair, and chain-of-thought reasoning steps.

Output: A single translated graph query string (Cypher for KGQA4MAT, SPARQL for QALD-9).

Scoring recipe

correct = 0
for q, gold_query in dataset:
    pred_query = model.generate(q, prompt_context)
    pred_res = execute_query(pred_query, backend)
    gold_res = execute_query(gold_query, backend)
    if set(pred_res) == set(gold_res):
        correct += 1
precision = correct / len(dataset)
recall = correct / len(dataset)
f1 = 2 * (1/precision + 1/recall)**-1
return f1

Common pitfalls

• Precision, recall, and F1 are mathematically identical because exactly one query is generated per test question.
• Evaluation only uses the English subset of QALD-9.
• For QALD-9, DBpedia ontology definitions are intentionally omitted from prompts under the assumption that ChatGPT already internalized them.

Evidence (verbatim from paper)

For each translated Cypher query, we execute it on the MOF-KG Neo4j database. We then compare the results to that retrieved by the correct Cypher query to see whether the translated query is correct. We measure the performance using the precision, recall, and F1-score in terms of the correctness of the translated queries: precision = # of correctly translated queries / # of total translated queries, recall = # of correctly translated queries / # of total correct queries, F1_score = 2(1/precision + 1/recall)^-1.

Citation

@misc{an2023kgqa4mat,
  title={Knowledge Graph Question Answering for Materials Science (KGQA4MAT): Developing Natural Language Interface for Metal-Organic Frameworks Knowledge Graph (MOF-KG) Using LLM},
  author={Yuan An et al. (2023)},
  year={2023},
  note={arXiv:2309.11361}
}

• arXiv: 2309.11361