Synthetic SQL Eval

synthetic-sql-eval

Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance — Wretblad et al. (2024) (arXiv:2408.04691, 2024)

What this evaluates

Evaluates how synthetic or human-generated column descriptions impact LLM performance on text-to-SQL tasks, and assesses the quality of LLM-generated descriptions across varying semantic difficulty levels.

Datasets

• BIRD-Bench — total 1534; splits: dev (1534); HF bird

Metrics

• Mean quality scores (primary) — range: numeric scale (1-5 or 1-10, exact range not specified)
  • Average rating assigned by human evaluators to generated column descriptions, aggregated across four semantic difficulty levels (Self-Evident, Context-Aided, Ambiguity-Prone, Domain-Dependent).
• Execution Accuracy — range: [0, 1]
  • Standard BIRD-Bench metric: proportion of predicted SQL queries that produce the exact same result set as the gold SQL query when executed on the target database.

Input / output format

Input: Database schema, specified column, example data rows, and generation instructions. For text-to-SQL evaluation: natural language question, database schema, and column descriptions (metadata).

Output: Natural language column description (for generation task) or SQL query (for text-to-SQL task).

Scoring recipe

# Description quality scoring:
scores = [human_rating(desc) for desc in generated_descriptions]
mean_quality_scores = sum(scores) / len(scores)

# Text-to-SQL scoring (standard BIRD protocol):
predicted_sql = model.generate(question, schema, descriptions)
execution_result = execute_sql(predicted_sql, database)
is_correct = execution_result == gold_result
execution_accuracy = sum(is_correct) / len(test_set)

Common pitfalls

• Assuming only 'perfect' descriptions improve performance; the paper shows 'superfluous' descriptions also significantly boost text-to-SQL accuracy.
• Overlooking the temperature difference: 0.7 for description generation vs 0.0 for text-to-SQL evaluation.
• Failing to account for the two-step ambiguity detection process, which requires human intervention for domain-dependent columns.

Evidence (verbatim from paper)

To study the usefulness of column descriptions for text-to-SQL systems, we first evaluated the same LLMs as in the description generation step in a zero-shot setting on the development set of BIRD-Bench, which contains 1534 natural language questions and SQL query pairs. ... Table 2: Mean quality scores for each model’s generated descriptions across the column difficulty levels in the dataset, as rated by the human evaluators.

Citation

@misc{wretblad2024synthetic,
  title={Synthetic SQL Column Descriptions and Their Impact on Text-to-SQL Performance},
  author={Wretblad et al. (2024)},
  year={2024},
  note={arXiv:2408.04691}
}

• arXiv: 2408.04691