Star SQL Eval

star-sql-eval

STaR-SQL: Self-Taught Reasoner for Text-to-SQL — He et al. (2025) (arXiv:2502.13550, 2025)

What this evaluates

Evaluates an LLM's ability to generate correct SQL queries from natural language questions over complex, multi-table database schemas. It probes the model's reasoning capabilities and schema generalization by requiring step-by-step rationales and testing on unseen databases.

Datasets

• Spider — total 10181; splits: train (8659), dev (1034)

Metrics

• execution accuracy (EX) (primary) — range: percent
  • Compares the execution output of the predicted SQL query with the ground truth SQL query on database instances. Returns 1 if the result sets match exactly, 0 otherwise. Averages over all instances.
• exact set match accuracy (EM) — range: percent
  • Treats each SQL clause as a set and compares the prediction to the reference query. A query is correct only if all components match. Ignores literal values.

Input / output format

Input: Natural language question and database schema (table names, column names, and types).

Output: A SQL query string, optionally preceded by step-by-step chain-of-thought reasoning rationales during inference.

Scoring recipe

def score(predictions, golds, db_instances):
    em_scores = []
    ex_scores = []
    for pred, gold in zip(predictions, golds):
        em_scores.append(1 if parse_clauses(pred) == parse_clauses(gold) else 0)
        pred_res = execute_sql(pred, db_instances)
        gold_res = execute_sql(gold, db_instances)
        ex_scores.append(1 if set(pred_res) == set(gold_res) else 0)
    return {'EM': sum(em_scores)/len(em_scores)*100, 'EX': sum(ex_scores)/len(ex_scores)*100}

Common pitfalls

• Exact set match (EM) ignores literal values and only checks structural clause equivalence, which can overestimate performance on queries with different constants.
• Execution accuracy (EX) requires a functional SQL engine and database instances; syntax errors in predictions will cause execution failures that must be handled gracefully.
• The Spider dataset splits databases non-overlappingly between train and dev, so models must generalize to entirely unseen schemas rather than memorizing table structures.

Evidence (verbatim from paper)

The performance of our models are evaluated using the official metrics of Spider (Zhong et al., [2020]): exact-setmatch accuracy (EM) and execution accuracy (EX). The exact-set-match accuracy (EM) treats each clause as a set and compares the prediction for each clause to its corresponding clause in the reference query. A predicted SQL query is considered correct only if all of its components match the ground truth. This metric does not take values into account. The execution accuracy (EX) compares the execution output of the predicted SQL query with that of the ground truth SQL query on some database instances.

Citation

@misc{he2025star_sql,
  title={STaR-SQL: Self-Taught Reasoner for Text-to-SQL},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2502.13550}
}

• arXiv: 2502.13550