Spider Eval

spider-eval

Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation — Guo et al. (2019) (arXiv:1905.08205, 2019)

What this evaluates

Evaluates a model's ability to translate natural language questions into correct SQL queries across diverse database domains. It probes schema linking, lexical matching, and complex query synthesis including joins, aggregations, and subqueries.

Datasets

• Spider — total ?; splits: train (-1), dev (-1); HF spider_dataset; repo https://github.com/taoyds/spider

Metrics

• exact matching accuracy (primary) — range: [0, 1]
  • A prediction is considered correct if and only if the generated SQL query is exactly identical to the ground truth SQL query, character-for-character. Otherwise, it is marked incorrect.

Input / output format

Input: Natural language question, database schema (table and column names), and schema linking results (entity types and matches).

Output: A SemQL intermediate representation query, which is deterministically converted to a SQL query for evaluation.

Scoring recipe

def exact_match_accuracy(predictions, gold):
    correct = 0
    for pred, gold_sql in zip(predictions, gold):
        if pred.strip() == gold_sql.strip():
            correct += 1
    return correct / len(gold)

Common pitfalls

• Exact matching requires character-for-character identity, so minor whitespace or case differences cause a penalty unless normalized.
• The evaluation is performed on the final SQL query derived from the intermediate SemQL representation, not on the SemQL itself.
• Cross-domain evaluation means the model must generalize to databases it has never seen during training, relying heavily on schema linking.

Evidence (verbatim from paper)

IRNet achieves 46.7% exact matching accuracy on Spider—19.5% higher than prior state-of-the-art—and sets the new leaderboard record, with BERT-augmented performance reaching 54.7%;

Citation

@misc{guo2019irnet,
  title={Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation},
  author={Guo et al. (2019)},
  year={2019},
  note={arXiv:1905.08205}
}

• arXiv: 1905.08205