Cross Domain Text To SQL Eval

cross-domain-text-to-sql-eval

Evaluating Cross-Domain Text-to-SQL Models and Benchmarks — Pourreza et al. (2023) (arXiv:2310.18538, 2023)

What this evaluates

Evaluates the reliability of cross-domain text-to-SQL benchmarks by exposing flaws in automated metrics like execution accuracy and exact set match, and by introducing human-in-the-loop validation to handle schema ambiguity and query equivalence.

Datasets

• Spider — total ?; splits: dev (1034)
• Spider-DK — total ?; splits: dev (-1)
• BIRD — total ?; splits: dev (-1)

Metrics

• Execution Accuracy (primary) — range: [0, 1]
  • The percentage of generated SQL queries that produce exactly the same result set as the ground truth query when executed against the database.
• Exact Set Match Accuracy — range: [0, 1]
  • The percentage of generated queries where the result set matches the ground truth result set exactly, ignoring row order.
• Human Accuracy — range: [0, 1]
  • The percentage of queries deemed correct by human annotators who evaluate semantic equivalence and schema compliance, resolving ambiguities that automated metrics miss.

Input / output format

Input: Natural language question, database schema, and database content.

Output: SQL query string.

Scoring recipe

def exec_acc(pred, gold, db):
    return execute_sql(pred, db) == execute_sql(gold, db)
def exact_set_match(pred, gold, db):
    return set(execute_sql(pred, db)) == set(execute_sql(gold, db))
def human_acc(preds, golds, annotators):
    correct = sum(1 for p, g in zip(preds, golds) if annotators_agree_correct(p, g))
    return correct / len(preds)

Common pitfalls

• Overreliance on strict row ordering or LIMIT 1 causes false negatives when multiple valid rows satisfy the condition.
• SQLite-specific syntax and loose typing mask standard SQL compliance issues, leading to false failures when validated against PostgreSQL.
• Schema ambiguity and incorrect database content assumptions create multiple valid SQL interpretations that automated metrics incorrectly reject.

Evidence (verbatim from paper)

Table 2 displays both the execution accuracy and the exact set match accuracy for the reference queries from the BIRD, Spider, and Spider-DK benchmarks after our modifications.

Citation

@misc{pourreza2023evaluating,
  title={Evaluating Cross-Domain Text-to-SQL Models and Benchmarks},
  author={Pourreza et al. (2023)},
  year={2023},
  note={arXiv:2310.18538}
}

• arXiv: 2310.18538