Kaggledbqa Eval

kaggledbqa-eval

KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers — Lee et al. (2021) (arXiv:2106.11455, 2021)

What this evaluates

Evaluates the zero-shot and few-shot generalization capability of text-to-SQL parsers on realistic, industrial-style database schemas with obscure column names and unrestricted natural language questions. It probes the model's ability to perform schema linking, constraint parsing, and SQL generation without extensive domain-specific training data.

Datasets

• KaggleDBQA — total ?; splits: train (87), test (185)

Metrics

• exact-match accuracy (primary) — range: percent
  • Exact string match between the predicted SQL and the gold SQL, evaluated using the Spider official evaluation scripts. The scripts normalize SQL by ignoring case, whitespace, and literal values before comparison.

Input / output format

Input: Natural language question paired with the target database schema (table and column names). In augmented settings, column descriptions are appended to column names.

Output: A single SQL query string.

Scoring recipe

def score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # Normalize SQL using Spider official script (ignore case, whitespace, values)
        pred_norm = normalize_sql(pred)
        gold_norm = normalize_sql(gold)
        if pred_norm == gold_norm:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Using raw string matching instead of the Spider official normalization script, which ignores case, whitespace, and literal values.
• Evaluating in a strict zero-shot setting without fine-tuning, which does not reflect the paper's recommended few-shot real-world evaluation protocol.
• Ignoring the impact of database documentation (column descriptions) and domain adaptation, which are critical for realistic performance on this dataset.

Evidence (verbatim from paper)

Throughout this paper, we use the same exact-match accuracy metric introduced by the Spider dataset. ... All numbers are the exact match accuracy evaluated by the Spider official scripts.

Citation

@misc{lee2021kaggledbqa,
  title={KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers},
  author={Lee et al. (2021)},
  year={2021},
  note={arXiv:2106.11455}
}

• arXiv: 2106.11455