Unite Eval

unite-eval

UNITE: A Unified Benchmark for Text-to-SQL Evaluation — Lan et al. (2023) (arXiv:2305.16265, 2023)

What this evaluates

Evaluates text-to-SQL models on compositional generalization, out-of-domain robustness, and schema-question alignment across 18 diverse datasets and 12 domains. It probes the model's ability to handle long-form query decomposition and cross-domain SQL pattern diversity.

Datasets

• UNITE — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Execution accuracy: the predicted SQL query is executed on the target database, and the output is compared against the gold execution result. A prediction is correct only if the execution results match exactly.

Input / output format

Input: Natural language question paired with the corresponding database schema.

Output: A single SQL query string.

Scoring recipe

def compute_accuracy(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        pred_result = execute_sql(pred_sql, db)
        gold_result = execute_sql(gold_sql, db)
        if pred_result == gold_result:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Prompt engineering and exemplar selection heavily influence in-context learning performance on out-of-domain data.
• Spider benchmark accuracy is a poor proxy for real-world application robustness.
• Constrained decoding is essential for generalizing to novel SQL structures and avoiding illegal generations.

Evidence (verbatim from paper)

We report zero-shot and 3-shot performance for Codex, by following prompt design in Rajkumar et al. (2022). For other fine-tuning based models, we report performance with the original public checkpoints trained on Spider dataset. We also re-train T5-3B and RASAT with our UNITE training set... Our main results are shown in Table 2. Firstly, these SOTA models have performance level 70% ~ 80% for Spider, but below 50% on our UNITE benchmark on average.

Citation

@misc{lan2023unite,
  title={UNITE: A Unified Benchmark for Text-to-SQL Evaluation},
  author={Lan et al. (2023)},
  year={2023},
  note={arXiv:2305.16265}
}

• arXiv: 2305.16265