SQL Exchange Eval

sql-exchange-eval

SQL-Exchange: Transforming SQL Queries Across Domains — Daviran et al. (2025) (arXiv:2508.07087, 2025)

What this evaluates

Evaluates an LLM's ability to translate SQL queries across different database schemas while preserving structural integrity and semantic meaning. It measures mapping success, structural fidelity, execution validity, and the semantic alignment between generated SQL and natural language questions.

Datasets

• BIRD — total 80; splits: train (69), dev (11)
• SPIDER — total 166; splits: train (146), dev (20)

Metrics

• Generation Success — range: percent
  • Percentage of queries for which the LLM generates a non-empty, parseable response instead of refusing or skipping the mapping.
• Structural Alignment (primary) — range: percent
  • Whether the mapped query preserves the structural skeleton of the source SQL, including keywords, control blocks (e.g., SELECT, JOIN, GROUP BY, subqueries), and logical structure. Table/column names, constants, and aliases are ignored. Basic comparison operators are normalized.
• Execution Validity — range: percent
  • Whether the generated SQL executes successfully on the target schema using SQLite.
• NL Meaningfulness — range: percent
  • LLM-as-judge evaluation (Gemini-2.0-flash) assessing whether the generated natural language question is clear, specific, and meaningful in the context of the target database schema.
• SQL-NL Alignment — range: percent
  • LLM-as-judge evaluation assessing whether the SQL query faithfully implements the generated question's intent.

Input / output format

Input: Task instruction, full source and target schema descriptions, target-side sample rows, and a batch of up to 10-20 source SQL queries.

Output: Mapped SQL query and corresponding natural language question.

Scoring recipe

def score_generation_success(response):
    return 1 if response is non-empty and parseable else 0

def score_structural_alignment(source_sql, target_sql):
    # Extract structural skeleton (keywords, control blocks, logical structure)
    # Ignore table/column names, constants, aliases
    # Normalize basic comparison operators (=, <, >)
    return 1 if skeletons_match(source_sql, target_sql) else 0

def score_execution_validity(target_sql, target_db):
    try:
        sqlite.execute(target_sql)
        return 1
    except Exception:
        return 0

# NL Meaningfulness & SQL-NL Alignment are scored via LLM-as-judge (Gemini-2.0-flash)
# Returns 1 if judged meaningful/aligned, else 0

Common pitfalls

• Overly strict structural preservation can hinder semantic consistency or lead to non-executable queries.
• Execution errors often stem from subtle schema mismatches (e.g., column/table references, quoting) rather than syntax errors.
• Schema incompatibilities may make direct translation infeasible, causing models to return null outputs instead of hallucinating mappings.

Evidence (verbatim from paper)

We first measure how reliably the model can produce mapped queries under the SQL-Exchange framework. This includes: Generation Success: Percentage of queries for which the LLM generates a non-empty, parseable response instead of refusing or skipping the mapping. Structural Alignment: Whether the mapped query preserves the structural skeleton of the source SQL, including keywords, control blocks (e.g., SELECT, JOIN, GROUP BY, subqueries), and logical structure. We ignore table and column names, constants, and optional aliasing via AS. Basic comparison operators (e.g., =, <, >) are normalized and treated equivalently. Execution Validity: Whether the generated SQL executes successfully on the target schema using SQLite.

Citation

@misc{daviran2025sqlexchange,
  title={SQL-Exchange: Transforming SQL Queries Across Domains},
  author={Daviran et al. (2025)},
  year={2025},
  note={arXiv:2508.07087}
}

• arXiv: 2508.07087