Patsql SQL Synthesis Eval

patsql-sql-synthesis-eval

PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns — Takenouchi et al. (2020) (arXiv:2010.05807, 2020)

What this evaluates

Evaluates the ability of program-by-example (PBE) systems to synthesize correct SQL queries from example input/output tables. It probes query generation accuracy, synthesis speed, and scalability to larger database schemas.

Datasets

• ase13 — total 28; splits: test (28); repo https://github.com/NAIST-SE/PATSQL
• so-top — total 57; splits: test (57); repo https://github.com/NAIST-SE/PATSQL
• so-dev — total 57; splits: test (57); repo https://github.com/NAIST-SE/PATSQL
• so-rec — total 51; splits: test (51); repo https://github.com/NAIST-SE/PATSQL
• kaggle — total 33; splits: test (33); repo https://github.com/NAIST-SE/PATSQL

Metrics

• solve_rate (primary) — range: percent
  • Number of benchmarks where the synthesized query matches the gold query semantics divided by the total number of benchmarks. Expressed as a count or percentage.
• synthesis_time — range: seconds
  • Wall-clock time in seconds required for the synthesis algorithm to produce a valid query or reach the timeout limit.

Input / output format

Input: Example input/output tables (I/O tables) containing specified columns and rows. Optional hints may be provided, including constants used in predicates and required aggregation/window functions.

Output: A synthesized SQL query string. For top-k variants (PATSQL_5, SCYTHE), a ranked list of up to 5 candidate queries.

Scoring recipe

def evaluate(predictions, gold_queries, io_tables):
    solved = 0
    for pred, gold in zip(predictions, gold_queries):
        # Verify semantic equivalence on the provided I/O tables
        if verify_semantics(pred, gold, io_tables):
            solved += 1
    return solved / len(gold_queries)

Common pitfalls

• SCYTHE and BaseLine require explicit hints for aggregation/window functions, while PATSQL does not, creating an unfair comparison if hinting protocols are not standardized.
• I/O tables for non-Kaggle benchmarks use extracted columns (avg 4.3) rather than full-scale schemas (avg 12.1), which artificially inflates scalability metrics for methods that struggle with large schemas.
• Semantic equivalence is verified manually by at least two SQL experts, which can introduce subjectivity or inconsistency across benchmarks.

Evidence (verbatim from paper)

For the benchmarks other than kaggle, PATSQL solved 134 benchmarks while SCYTHE solved 128 (see Table 2). In particular, PATSQL succeeded in synthesizing 20 queries from I/O tables with a larger number of cells that SCYTHE failed to solve due to scalability issues.

Citation

@misc{takenouchi2020patsql,
  title={PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns},
  author={Takenouchi et al. (2020)},
  year={2020},
  note={arXiv:2010.05807}
}

• arXiv: 2010.05807