qhjqhj00 xiyansql-eval
A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL — Gao et al. (2024) (arXiv:2411.08599, 2024)
What this evaluates
Evaluates the ability of text-to-SQL models to generate correct SQL or GQL queries for natural language questions across relational and graph databases. It measures execution accuracy by comparing the runtime results of generated queries against reference queries on specific database instances.
Datasets
- Spider — total 1981; splits: test (1981)
- Bird — total 1534; splits: dev (1534)
- SQL-Eval — total 304; splits: test (304)
- NL2GQL — total 288; splits: test (288)
Metrics
Execution Accuracy (EX)(primary) — range: percent- Execution Accuracy (EX) compares the results of a predicted SQL query and a reference SQL query executed on a specific database instance. A prediction is marked correct if the execution results match exactly.
Input / output format
Input: Natural language question and database schema (provided as M-Schema, DDL, or MAC-SQL representation including table/column names, data types, primary keys, and example values).
Output: A single SQL or GQL query string.
Scoring recipe
correct = 0
for instance in dataset:
pred_res = execute_sql(instance.pred_sql, instance.db_id)
ref_res = execute_sql(instance.ref_sql, instance.db_id)
if pred_res == ref_res:
correct += 1
return (correct / len(dataset)) * 100
Common pitfalls
- SQL-Eval provides multiple reference SQL queries per question; the paper uses only the first one as ground truth for metric computation.
- Bird benchmark test set is not publicly available, so evaluations are conducted on the development set instead.
- Schema representation format (M-Schema vs DDL vs MAC-SQL) significantly impacts model performance and must be standardized for fair comparison.
Evidence (verbatim from paper)
We use Execution Accuracy (EX) to access the effectiveness of the generated SQL queries. EX compares the results of a predicted SQL query and a reference SQL query executed on a specific database instance.
Citation
@misc{gao2024xiyansql,
title={A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL},
author={Gao et al. (2024)},
year={2024},
note={arXiv:2411.08599}
}
- arXiv: 2411.08599