Bird Bench Eval

bird-bench-eval

Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark — Wretblad et al. (2024) (arXiv:2402.12243, 2024)

What this evaluates

Evaluates the capability of large language models to generate correct SQL queries from natural language questions. It specifically probes how annotation noise and errors in benchmark datasets affect model performance and reliability.

Datasets

• BIRD-Bench — total 12751; splits: dev (106)

Metrics

• accuracy (primary) — range: [0, 1]
  • The fraction of questions for which the model's generated SQL query yields the correct result or exactly matches the gold query.

Input / output format

Input: Natural language question, database schema (provided as SQL CREATE TABLE statements), and database context.

Output: A single SQL query string.

Scoring recipe

correct = 0
for pred_sql, gold_sql in zip(predictions, gold_queries):
    if pred_sql == gold_sql or execute(pred_sql) == execute(gold_sql):
        correct += 1
return correct / len(predictions)

Common pitfalls

• Noise in gold SQL queries propagates false answers, causing models to be incorrectly penalized for logically correct outputs.
• Question ambiguity and uneven domain-specific noise distribution can skew performance metrics, making some domains appear artificially harder or easier.
• Advanced prompting methods may overfit to noisy annotations rather than learning robust SQL generation capabilities.

Evidence (verbatim from paper)

Figure 2: Accuracy of various models on Bird-Bench’s financial domain. Models are evaluated on the original data (left), corrected SQL queries (middle), and corrected SQL queries and corrected noisy questions.

Citation

@misc{wretblad2024birdnoise,
  title={Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark},
  author={Wretblad et al. (2024)},
  year={2024},
  note={arXiv:2402.12243}
}

• arXiv: 2402.12243