SQL Synth Eval

sql-synth-eval

Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation — Wang et al. (2025) (arXiv:2511.13590, 2025)

What this evaluates

This benchmark evaluates a model's ability to translate natural language questions into correct SQL queries across diverse, real-world database schemas. It specifically probes the model's capacity to handle complex, multi-operation statements and cross-domain syntax structures that are often underrepresented in traditional benchmarks.

Datasets

• SQL-Synth — total ?; splits: train (-1), test (-1)

Metrics

• execution accuracy (EX) (primary) — range: [0, 1]
  • Measures whether the predicted SQL query produces the exact same execution results as the corresponding gold SQL query when run against the target database.
• Data Quality Score — range: [0, 1]
  • Weighted average of LLM-as-a-judge ratings: (N_e1 + N_g0.75 + N_a0.5 + N_p0.25) / (N_e + N_g + N_a + N_p), where N_x is the count of samples rated Excellent, Good, Average, or Poor across question, SQL, and result aspects.

Input / output format

Input: Natural language question paired with the target database schema (table names, columns, and relationships).

Output: A single SQL query string.

Scoring recipe

def compute_ex(predictions, golds, databases):
    correct = 0
    for pred, gold, db in zip(predictions, golds, databases):
        try:
            pred_res = execute_query(pred, db)
            gold_res = execute_query(gold, db)
            if sets_equal(pred_res, gold_res):
                correct += 1
        except Exception:
            pass
    return correct / len(predictions)

Common pitfalls

• Execution accuracy requires a functional database engine to run queries; string matching or syntax checking alone will yield incorrect scores.
• The dataset intentionally includes complex, multi-operation queries across cross-domain schemas, so models may fail on schema alignment or syntax if evaluated without proper fine-tuning or context handling.
• The quality evaluation relies on GPT-4o as a judge, which may introduce bias or inconsistency compared to ground-truth execution metrics.

Evidence (verbatim from paper)

Following previous work, we use execution accuracy (EX) as the evaluation metric, which measures whether the predicted SQL query produces the same execution results as the corresponding gold SQL query.

Citation

@misc{wang2025sqlsynth,
  title={Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2511.13590}
}

• arXiv: 2511.13590