Dr Spider Eval

dr-spider-eval

Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness — Chang et al. (2023) (arXiv:2301.08881, 2023)

What this evaluates

Evaluates the robustness of text-to-SQL models against semantic-preserving and semantic-changing perturbations across database schemas, natural language questions, and SQL queries. It measures how well models maintain execution accuracy when inputs are altered, revealing architectural vulnerabilities in entity linking, decoder design, and value prediction.

Datasets

• Dr.Spider — total ?; splits: test (-1); repo https://github.com/awslabs/diagnostic-robustness-text-to-sql

Metrics

• exact set match (EM) — range: [0, 1]
  • Measures a predicted SQL with a gold SQL on each SQL clause ignoring values.
• execution accuracy (EX) (primary) — range: [0, 1]
  • Compares the denotation answers from a predicted SQL and a gold SQL to evaluate correctness of SQL values.
• pre-perturbation accuracy — range: [0, 1]
  • The accuracy on pre-perturbation (original) data.
• post-perturbation accuracy — range: [0, 1]
  • The accuracy on post-perturbation data (also called absolute robustness accuracy).
• relative robustness accuracy — range: [0, 1]
  • The ratio of the number of correct predictions on both parallel pre-perturbation and post-perturbation data over the number of correct predictions on pre-perturbation data.

Input / output format

Input: Natural language question (NLQ) and database schema (table names, column names, and types).

Output: A single SQL query string.

Scoring recipe

def compute_metrics(pred_sqls, gold_sqls, dbs, is_perturbed=False):
    em = [exact_set_match(p, g) for p, g in zip(pred_sqls, gold_sqls)]
    ex = [execution_accuracy(p, g, db) for p, g, db in zip(pred_sqls, gold_sqls, dbs)]
    if not is_perturbed:
        pre_acc = sum(ex) / len(ex)
        correct_pre = [i for i, v in enumerate(ex) if v]
        return {'pre_acc': pre_acc, 'EM': sum(em)/len(em), 'EX': sum(ex)/len(ex)}
    else:
        post_acc = sum(ex) / len(ex)
        correct_post = [i for i, v in enumerate(ex) if v]
        both_correct = sum(1 for i in correct_pre if i in correct_post)
        rel_acc = both_correct / len(correct_pre) if correct_pre else 0
        return {'post_acc': post_acc, 'relative_acc': rel_acc}

Common pitfalls

• EM only checks clause structure ignoring literal values, while EX checks actual database results; confusing them leads to incorrect robustness claims.
• Relative robustness accuracy is a conditional ratio (correct on both / correct on pre), not a simple accuracy difference or ratio of overall scores.
• GraPPa omits value predictions, so EM is reported for it but EX is the primary metric for other models; comparing them directly on EX is invalid.

Evidence (verbatim from paper)

We follow the Spider Benchmark to use the two SQL evaluation metrics to evaluate the correctness of a predicted SQL with a gold SQL: exact set match (EM) measures a predicted SQL with a gold SQL on each SQL clause ignoring values; execution accuracy (EX) compares the denotation answers from a predicted SQL and a gold SQL. We use EX as our main evaluation metric as it evaluates the correctness of SQL values, which is an important part of model robustness, while we only report EM for Grappa which omits value predictions. To evaluate robustness, we report 3 metrics: (1) pre-perturbation accuracy: the accuracy on pre-perturbation data, (2) post-perturbation accuracy (or absolute robustness accuracy): the accuracy on post-perturbation data, (3) relative robustness accuracy: The ratio of the number of correct predictions on both parallel pre-perturbation and post-perturbation data over the number of correct predictions on pre-perturbation data.

Citation

@misc{chang2023drspider,
  title={Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness},
  author={Chang et al. (2023)},
  year={2023},
  note={arXiv:2301.08881}
}

• arXiv: 2301.08881