Footballdb Eval

footballdb-eval

Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries — Fürst et al. (2024) (arXiv:2402.08349, 2024)

What this evaluates

This benchmark evaluates the robustness and accuracy of Text-to-SQL systems when translating natural language questions into SQL queries across different database schema designs. It probes how data model complexity, training data size, and language model scale impact execution accuracy on real-world user queries.

Datasets

• FootballDB — total 1200; splits: train (300), test (100)

Metrics

• exact execution matching (EX) (primary) — range: percent
  • Fraction of questions within the evaluation set where the outcomes of the predicted and ground-truth SQL queries yield identical results relative to the total number of queries.

Input / output format

Input: Natural language question, database schema (with or without primary/foreign key constraints depending on the system), and optionally database content or string-normalized text.

Output: A single SQL query string.

Scoring recipe

correct = 0
for q, sql_gold in test_set:
    sql_pred = model.generate(q, schema)
    if execute(sql_pred) == execute(sql_gold):
        correct += 1
return (correct / len(test_set)) * 100

Common pitfalls

• The Spider hardness metric underestimates query complexity for this dataset, which features significantly more joins and SQL tokens.
• The standard SemEval Test Suite evaluation script fails to parse many queries due to built-in SQL parser limitations, necessitating exact execution matching instead.
• Few-shot LLM performance shows high variance depending on the specific random samples chosen for the prompt.

Evidence (verbatim from paper)

Therefore, we apply exact execution matching (EX), also known as result matching as the accuracy metrics instead of exact SQL component matching as in the test suite evaluation. EX denotes the fraction of questions within the evaluation set, where the outcomes of both the predicted and ground-truth queries yield identical results relative to the total number of queries.

Citation

@misc{fuerst2024footballdb,
  title={Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries},
  author={Fürst et al. (2024)},
  year={2024},
  note={arXiv:2402.08349}
}

• arXiv: 2402.08349