Wikisql Eval

wikisql-eval

Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning — Zhong et al. (2017) (arXiv:1709.00103, 2017)

What this evaluates

Evaluates a model's ability to translate natural language questions into executable SQL queries against a given database schema. It probes semantic parsing, table understanding, and the generation of syntactically and semantically correct structured queries.

Datasets

• WikiSQL — total 80654; splits: dev (-1), test (-1); repo https://github.com/salesforce/WikiSQL

Metrics

• Acc_lf — range: percent
  • Logical Form Accuracy: exact string match between the generated SQL query and the ground truth query.
• Acc_ex (primary) — range: percent
  • Execution Accuracy: the generated query is executed on the database, and its result set is compared to the result set of the ground truth query. Returns 1 if they match, 0 otherwise.

Input / output format

Input: Natural language question paired with the database table schema (column names, types, and table name).

Output: A valid SQL SELECT query string.

Scoring recipe

def score_acc_ex(predictions, golds, db_engine):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            pred_res = db_engine.execute(pred)
            gold_res = db_engine.execute(gold)
            if set(pred_res) == set(gold_res):
                correct += 1
        except:
            pass
    return correct / len(predictions)

def score_acc_lf(predictions, golds):
    return sum(1 for p, g in zip(predictions, golds) if p == g) / len(predictions)

Common pitfalls

• Execution accuracy is order-invariant for WHERE conditions, so models can generate syntactically different but semantically equivalent queries (e.g., swapped AND clauses) and still get full credit.
• Column names with multiple tokens (e.g., 'Miles (km)') are prone to tokenization errors, leading to invalid queries or mismatched column references.
• Rare words in conditions are often hallucinated or truncated by baselines, hurting logical form accuracy even if execution accuracy remains high.

Evidence (verbatim from paper)

Dev Acc_lf | Dev Acc_ex | Test Acc_lf | Test Acc_ex

Citation

@misc{zhong2017seq2sql,
  title={Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning},
  author={Zhong et al. (2017)},
  year={2017},
  note={arXiv:1709.00103}
}

• arXiv: 1709.00103