Sparc Cg Eval

sparc-cg-eval

Exploring the Compositional Generalization in Context Dependent Text-to-SQL Parsing — Liu et al. (2023) (arXiv:2306.04480, 2023)

What this evaluates

This benchmark probes a model's ability to perform compositional generalization in context-dependent Text-to-SQL. It evaluates whether models can correctly combine previously seen SQL query structures with novel modification patterns (e.g., new WHERE or ORDER BY clauses) in multi-turn dialogues.

Datasets

• SPARC-CG — total ?; splits: Dev (-1), Non-CG (-1), CG (-1); repo https://github.com/THU-BPM/CD-Text2SQL-CG

Metrics

• question match (QM) (primary) — range: percent
  • Decomposes predicted and gold SQL queries into components (e.g., SELECT, WHERE, GROUP BY). Computes the exact set matching score for each component and averages them across all questions to yield the final score.

Input / output format

Input: A multi-turn dialogue context consisting of previous user questions and the corresponding SQL queries, followed by the current user question.

Output: A single SQL query string.

Scoring recipe

def compute_qm(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        pred_comps = decompose_sql(pred)
        gold_comps = decompose_sql(gold)
        comp_scores = [exact_match(pred_comps[c], gold_comps[c]) for c in components]
        scores.append(mean(comp_scores))
    return mean(scores)

Common pitfalls

• The original SPARC/CoSQL 'Dev' sets contain many context-independent questions, inflating base performance compared to the strictly context-dependent Non-CG and CG splits.
• QM evaluates component-level exact matching, which may penalize syntactically equivalent but structurally different SQL queries (e.g., different join orders or alias usage).
• Models frequently conflate context information with modification information, making error attribution difficult without detailed component-level analysis.

Evidence (verbatim from paper)

We mainly use the question match (QM) (Yu et al., 2019b) as our evaluation metric, which is the exact set matching score (Yu et al., 2018) over all questions. The exact set matching score decomposes predicted queries into SQL components such as SELECT and WHERE and then computes scores for each component.

Citation

@misc{liu2023exploring,
  title={Exploring the Compositional Generalization in Context Dependent Text-to-SQL Parsing},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.04480}
}

• arXiv: 2306.04480