Xiyan SQL Eval

xiyan-sql-eval

XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL — Liu et al. (2025) (arXiv:2507.04701, 2025)

What this evaluates

Evaluates a Text-to-SQL framework's ability to generate correct and efficient SQL queries from natural language questions across complex, cross-domain databases. It probes schema filtering, multi-generator candidate creation, and selection robustness.

Datasets

• BIRD — total 12751; splits: train (9428), dev (1534), test (1789)
• Spider — total 10181; splits: train (8659), dev (1034), test (2147)

Metrics

• Execution Accuracy (EX) (primary) — range: percent
  • The percentage of predicted SQL queries whose execution results exactly match the ground-truth SQL query execution results. Calculated as (correct predictions / total predictions) * 100. Note that execution rules differ slightly between BIRD and Spider.
• Reward-based Valid Efficiency Score (R-VES) — range: percent
  • An adjusted metric introduced in the latest BIRD version that evaluates both the accuracy and the runtime efficiency of correctly predicted SQL queries.

Input / output format

Input: Natural language question and database schema (represented in M-schema format, optionally filtered by a schema filter module).

Output: A single SQL query per question.

Scoring recipe

def compute_ex(predictions, gold_sqls, databases):
    correct = 0
    for pred, gold, db in zip(predictions, gold_sqls, databases):
        if execute(pred, db) == execute(gold, db):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Execution Accuracy (EX) calculation differs slightly between the BIRD and Spider benchmarks, so implementations must follow each benchmark's specific execution rules.
• R-VES is exclusive to the BIRD benchmark and evaluates both correctness and runtime efficiency, not just logical equivalence.
• Omitting the schema filter module causes a ~1.24% drop in EX, so full-schema baselines are not directly comparable to filtered ones.

Evidence (verbatim from paper)

Following the previous studies, we use Execution Accuracy (EX) to evaluate the performance of Text-to-SQL methods. Since an SQL query can be expressed in various forms, EX is used to assess the validity of the predicted SQL query and to determine whether the execution results are consistent with the ground-truth SQL query. It should be noted that there are slight differences in the calculation of the Execution Accuracy between the BIRD and Spider benchmarks; however, their overall objective remains consistent.

Citation

@misc{liu2025xiyansql,
  title={XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2507.04701}
}

• arXiv: 2507.04701