Kgqa Eval

kgqa-eval

ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots — Omar et al. (2023) (arXiv:2302.06466, 2023)

What this evaluates

This benchmark evaluates the ability of conversational AI models and traditional knowledge graph question-answering systems to accurately answer natural language questions over structured knowledge graphs. It probes factual grounding, recall on exhaustive lists, robustness to linguistic variations, and determinism across general and academic domains.

Datasets

• QALD-9 — total 150; splits: test (150)
• YAGO — total 100; splits: test (100)
• DBLP — total 100; splits: test (100)
• MAG — total 100; splits: test (100)

Metrics

• Precision — range: percent
  • The proportion of generated answers that are correct out of all non-empty answers provided. Calculated as TP / (TP + FP).
• Recall — range: percent
  • The proportion of ground truth answers successfully retrieved out of all questions. Calculated as TP / (TP + FN).
• Micro F1 score (primary) — range: percent
  • The harmonic mean of precision and recall, computed as 2 * (Precision * Recall) / (Precision + Recall). Aggregated across all questions in the benchmark.

Input / output format

Input: English natural language question, optionally preceded by a system prompt (e.g., 'List out all...' or Excel formatting instructions).

Output: Textual answer, optionally structured as an Excel table or enumerated list depending on the prompt variation.

Scoring recipe

def evaluate(predictions, gold):
    correct = 0
    wrong = 0
    no_answer = 0
    for pred, gold_ans in zip(predictions, gold):
        if pred == 'No answer':
            no_answer += 1
        elif matches_gold(pred, gold_ans):
            correct += 1
        else:
            wrong += 1
    precision = correct / (correct + wrong) if (correct + wrong) > 0 else 0
    recall = correct / (correct + no_answer + wrong) if (correct + no_answer + wrong) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• ChatGPT's default mode severely underestimates recall on list questions unless explicitly prompted with Excel or Follow-up variations.
• Language models are non-deterministic; evaluating on a single run yields unreliable metrics, requiring multiple runs or majority voting to assess true capability.
• Ground truth answers are derived by executing SPARQL queries, so evaluation assumes the SPARQL execution results are the absolute reference, ignoring potential query formulation errors in the benchmark.

Evidence (verbatim from paper)

answers are classified into three categories: (i) Correct, where ChatGPT provides correct answers with respect to the golden truth; (ii) Wrong, where ChatGPT provides answers that do not match the golden truth; (iii) No answer, where ChatGPT is unable to produce any answer for the given question. Table 1 summarizes the precision, recall and micro F1 score for each competitor in each benchmark.

Citation

@misc{omar2023chatgpt,
  title={ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots},
  author={Omar et al. (2023)},
  year={2023},
  note={arXiv:2302.06466}
}

• arXiv: 2302.06466