Spider Syn Eval

spider-syn-eval

Towards Robustness of Text-to-SQL Models against Synonym Substitution — Gan et al. (2021) (arXiv:2106.01065, 2021)

What this evaluates

This benchmark evaluates the robustness of text-to-SQL models when natural language questions contain real-world synonyms replacing schema-related terms. It probes whether models rely on rigid lexical matching or can generalize to paraphrased queries while preserving the underlying database schema and target SQL query.

Datasets

• Spider — total 8034; splits: train (7000), dev (1034)
• Spider-Syn — total 8034; splits: train (7000), dev (1034)

Metrics

• exact matching accuracy (primary) — range: percent
  • Calculated as the percentage of instances where the model's predicted SQL query exactly matches the ground-truth SQL query. Follows the official Spider evaluation convention which typically normalizes whitespace and case before comparison.

Input / output format

Input: A natural language question paired with the corresponding database schema (table names, column names, data types, and foreign key relationships).

Output: A single SQL query string.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_sqls):
    if pred.strip() == gold.strip():
        correct += 1
accuracy = (correct / len(predictions)) * 100
return accuracy

Common pitfalls

• The Spider test set is not publicly accessible, forcing all evaluations to rely on the development set which may not capture full generalization.
• Models frequently fail on synonym-substituted questions due to over-reliance on exact lexical matching between question tokens and schema items, rather than logical reasoning.
• Adversarial training or multi-annotation defenses can show high accuracy on specific attack sets but may degrade performance on clean, in-distribution data.

Evidence (verbatim from paper)

Models are evaluated using the official exact matching accuracy metric of Spider. Both Spider and Spider-Syn contain 7000 training and 1034 development samples respectively, where there are 146 databases for training and 20 for development.

Citation

@misc{gan2021towards,
  title={Towards Robustness of Text-to-SQL Models against Synonym Substitution},
  author={Gan et al. (2021)},
  year={2021},
  note={arXiv:2106.01065}
}

• arXiv: 2106.01065