Ezsql SQL To Text Eval

ezsql-sql-to-text-eval

EzSQL: An SQL intermediate representation for improving SQL-to-text Generation — Bhardwaj et al. (2024) (arXiv:2411.18923, 2024)

What this evaluates

Evaluates a model's capability to generate fluent natural language descriptions from SQL queries (SQL-to-text) and measures how well the generated text can augment training data for Text-to-SQL parsers.

Datasets

• WikiSQL — total 87726; splits: train (61297), dev (9145), test (17284)
• Spider — total 10181; splits: train (7000), dev (1034), test (2147)

Metrics

• BLEU-4 (primary) — range: [0, 100]
  • Standard BLEU-4 metric computing the geometric mean of 1-gram to 4-gram precisions with a brevity penalty. The paper reports scores scaled by 100.
• Exact Match — range: [0, 100]
  • Binary indicator that is 1 if the predicted SQL string exactly matches the gold SQL string, and 0 otherwise. Averaged across the test set.

Input / output format

Input: SQL query string (transformed via EzSQL intermediate representation) for SQL-to-text generation; natural language question for Text-to-SQL parsing.

Output: Natural language sentence describing the SQL query; SQL query string.

Scoring recipe

def compute_metrics(predictions, golds):
    bleu_scores = [nltk.bleu4([g.split()], [p.split()]) * 100 for p, g in zip(predictions, golds)]
    exact_matches = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    return {
        'BLEU-4': sum(bleu_scores) / len(bleu_scores),
        'Exact Match': sum(exact_matches) / len(exact_matches) * 100
    }

Common pitfalls

• BLEU-4 measures surface n-gram overlap and does not capture semantic equivalence or SQL execution correctness.
• Spider uses a database-specific split (146/20/40 databases) rather than random instance splits, which significantly impacts generalization evaluation compared to standard benchmarks.

Evidence (verbatim from paper)

We evaluate our model on two datasets, WikiSQL and Spider. WikiSQL consists of a corpus of 87,726 hand-annotated SQL queries and natural language question pairs. These SQL queries are further split into training (61,297 examples), development (9,145 examples), and test sets (17,284 examples). Spider consists of 7000, 1034, and 2147 samples for training, development, and testing, respectively... The BLEU-4 score is used as the automatic evaluation metric for this task.

Citation

@misc{bhardwaj2024ezsql,
  title={EzSQL: An SQL intermediate representation for improving SQL-to-text Generation},
  author={Bhardwaj et al. (2024)},
  year={2024},
  note={arXiv:2411.18923}
}

• arXiv: 2411.18923