Spider Cosql Eval

spider-cosql-eval

PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models — Scholak et al. (2021) (arXiv:2109.05093, 2021)

What this evaluates

Evaluates the text-to-SQL and dialog state tracking capabilities of language models by measuring how accurately they generate syntactically and semantically valid SQL queries from natural language questions, with and without constrained auto-regressive decoding.

Datasets

• Spider — total 7000; splits: train (7000), dev (-1), test (-1)
• CoSQL — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• exact-set-match accuracy (primary) — range: percent
  • Compares predicted and ground-truth SQL queries by parsing both into a normalized data structure. Ignores literal query values but can decrease under semantic-preserving rewrites.
• execution accuracy — range: percent
  • Compares the results of executing the predicted SQL query against the ground-truth SQL query on the provided database contents. Sensitive to literal values but has a high false positive rate.
• test-suite execution accuracy — range: percent
  • Extends execution accuracy by testing against multiple database instances per SQL schema, optimized to reduce false positives and approximate semantic accuracy.
• question match accuracy — range: percent
  • Exact-set-match accuracy applied to individual questions within a dialog interaction.
• interaction match accuracy — range: percent
  • Joint exact-set-match accuracy over all questions in a multi-turn dialog interaction.

Input / output format

Input: Natural language question concatenated with database schema (and previous dialog questions in reverse chronological order for CoSQL), truncated to 512 tokens. Keywords and identifiers are lowercased.

Output: A single SQL query string, with keywords and identifiers converted to lowercase.

Scoring recipe

def score(predictions, golds, db_contents):
    em_scores, ex_scores = [], []
    for pred, gold in zip(predictions, golds):
        em_scores.append(normalize_sql(pred) == normalize_sql(gold))
        pred_res = execute_sql(pred, db_contents)
        gold_res = execute_sql(gold, db_contents)
        ex_scores.append(pred_res == gold_res)
    return {
        'exact_set_match_accuracy': sum(em_scores) / len(em_scores),
        'execution_accuracy': sum(ex_scores) / len(ex_scores)
    }

Common pitfalls

• Execution accuracy can yield false positives when semantically different SQL queries produce identical results on a specific database instance.
• Exact-set-match accuracy is sensitive to literal query values and semantic-preserving rewrites, potentially underestimating correctness.
• Constrained decoding must be applied incrementally during beam search; applying it only post-hoc significantly reduces effectiveness.

Evidence (verbatim from paper)

On Spider, we determine model performance based on three metrics: exact-set-match accuracy, execution accuracy, and test-suite execution accuracy (Zhong et al., 2020). Exact-set-match accuracy compares the predicted and the ground-truth SQL query by parsing both into a normalized data structure. This comparison is not sensitive to literal query values and can decrease under semantic-preserving SQL query rewriting. Execution accuracy compares the results of executing the predicted and ground-truth SQL queries on the database contents shipped with the Spider dataset.

Citation

@misc{scholak2021picard,
  title={PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models},
  author={Scholak et al. (2021)},
  year={2021},
  note={arXiv:2109.05093}
}

• arXiv: 2109.05093