Sede Eval

sede-eval

Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data — Hazoom et al. (2021) (arXiv:2106.05006, 2021)

What this evaluates

Evaluates text-to-SQL models on naturally occurring, under-specified user queries from Stack Exchange. Probes the model's ability to handle real-world ambiguity, nested subqueries, parameterized queries, and domain-specific schema knowledge without relying on perfectly specified instructions.

Datasets

• SEDE — total 12023; splits: train (-1), val (-1), test (-1); repo https://github.com/hirupert/sede

Metrics

• PCM-F1 (primary) — range: [0, 1]
  • F1 score computed over partially matched SQL clauses between predicted and gold queries. Designed to relax exact-match strictness to capture structural similarity in real-world, under-specified queries.
• PCM-EM — range: [0, 1]
  • Binary accuracy that returns 1 if and only if PCM-F1 equals 1, otherwise 0.

Input / output format

Input: Natural language utterance (query title), optionally followed by a separator token and a serialized schema listing tables and columns.

Output: A single SQL query string.

Scoring recipe

def compute_pcm_f1(pred_sql, gold_sql):
    pred_clauses = extract_clauses(pred_sql)
    gold_clauses = extract_clauses(gold_sql)
    tp = len(match(pred_clauses, gold_clauses))
    precision = tp / len(pred_clauses) if pred_clauses else 0
    recall = tp / len(gold_clauses) if gold_clauses else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

def compute_pcm_em(pred_sql, gold_sql):
    return 1.0 if compute_pcm_f1(pred_sql, gold_sql) == 1.0 else 0.0

Common pitfalls

• Assuming exact string match (EM) is sufficient; real-world queries often differ in column selection or ordering despite high semantic overlap, making PCM-F1 necessary.
• Forgetting that the dataset uses a single schema seen during training, so encoding the schema during inference does not significantly boost performance.
• Including query descriptions in the input; the protocol explicitly uses only titles, which are often under-specified.

Evidence (verbatim from paper)

For each experiment we measure PCM-F1 together with a modified version of it, PCM-EM (PCM exact match), that returns an accuracy of 1 for a given prediction if and only if the PCM-F1 value for that prediction is 1.

Citation

@misc{hazoom2021sede,
  title={Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data},
  author={Hazoom et al. (2021)},
  year={2021},
  note={arXiv:2106.05006}
}

• arXiv: 2106.05006