Enterprise SQL Kg QA Eval

enterprise-sql-kg-qa-eval

A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases — Sequeda et al. (2023) (arXiv:2311.07509, 2023)

What this evaluates

Evaluates large language models' ability to generate correct SQL or SPARQL queries for natural language questions over an enterprise insurance database. It measures how well zero-shot prompting with raw schema versus knowledge graph augmentation improves factual grounding and query execution accuracy.

Datasets

• Enterprise SQL & KG QA Benchmark — total 43; splits: test (43); repo https://github.com/datadotworld/cwd-benchmark-data

Metrics

• execution accuracy (primary) — range: [0, 1]
  • Binary score: 1 if the result of the generated query matches the result of the reference query exactly, 0 otherwise. Reported as the average across all 43 questions.

Input / output format

Input: Natural language question combined with either the SQL DDL schema (for SQL generation) or the OWL ontology in TTL format (for SPARQL generation).

Output: A single SQL or SPARQL query string, with no explanations or markdown formatting, intended to be run verbatim.

Scoring recipe

def compute_execution_accuracy(generated_queries, reference_queries, data_loader):
    correct = 0
    for gen_q, ref_q in zip(generated_queries, reference_queries):
        try:
            gen_res = execute_query(gen_q, data_loader)
            ref_res = execute_query(ref_q, data_loader)
            if dataframes_match(gen_res, ref_res):
                correct += 1
        except Exception:
            pass
    return correct / len(reference_queries)

Common pitfalls

• Timeouts, network failures, or syntactically invalid queries are all treated as execution failures (score 0).
• Multiple reference queries per question must yield identical results; otherwise, the benchmark setup is invalid.
• The comparison relies on exact DataFrame equality, so schema/column ordering differences may cause false negatives.

Evidence (verbatim from paper)

The point of the experimental setup is to gather the data needed to compute execution accuracy (from which we can compute the derivative metrics of Overall Execution Accuracy and Average Overall Execution Accuracy). The basis of Execution Accuracy is to ask the Question Answering system to generate a query, execute that query, and compare the results of the generate query to the results given by the corresponding reference query.

Citation

@misc{sequeda2023enterprise,
  title={A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases},
  author={Sequeda et al. (2023)},
  year={2023},
  note={arXiv:2311.07509}
}

• arXiv: 2311.07509