Sparc Eval

sparc-eval

SParC: Cross-Domain Semantic Parsing in Context — Yu et al. (2019) (arXiv:1906.02285, 2019)

What this evaluates

Probes cross-domain semantic parsing in context by requiring models to generate sequential SQL queries across multiple conversational turns. It evaluates the ability to maintain state, handle thematic evolution, and generalize to unseen databases while correctly resolving contextual dependencies.

Datasets

• SParC — total ?; splits: dev (-1), test (-1)

Metrics

• question match (primary) — range: percent
  • Exact set matching score over all questions. A score of 1 is assigned to a question only if all predicted SQL clauses (SELECT, WHERE, GROUP BY, ORDER BY) exactly match the gold clauses via set comparison.
• interaction match — range: percent
  • Exact set matching score over all interactions. A score of 1 is assigned to an interaction only if every question within that interaction achieves an exact set match.

Input / output format

Input: Natural language question (or turn), database schema, and conversation history (previous questions).

Output: SQL query string.

Scoring recipe

def clause_exact_match(gold_sql, pred_sql):
    gold_clauses = decompose(gold_sql)  # SELECT, WHERE, GROUP BY, ORDER BY
    pred_clauses = decompose(pred_sql)
    return all(set(g) == set(p) for g, p in zip(gold_clauses, pred_clauses))

def evaluate(gold_data, pred_data):
    q_accs, i_accs = [], []
    for gold_int, pred_int in zip(gold_data, pred_data):
        int_correct = True
        for q_g, q_p in zip(gold_int, pred_int):
            match = clause_exact_match(q_g, q_p)
            q_accs.append(1.0 if match else 0.0)
            if not match: int_correct = False
        i_accs.append(1.0 if int_correct else 0.0)
    return sum(q_accs)/len(q_accs), sum(i_accs)/len(i_accs)

Common pitfalls

• Evaluating SQL correctness via simple string matching instead of clause-level set matching, which penalizes syntactically equivalent but differently ordered queries.
• Treating each turn as independent rather than accounting for sequential dependency and error propagation across turns.
• Confusing question match (per-question accuracy) with interaction match (requires all questions in a sequence to be correct), leading to misreported model capabilities.

Evidence (verbatim from paper)

Following Yu et al. (2018c), we use the exact set match metric to compute the accuracy between gold and predicted SQL answers. Instead of simply employing string match, Yu et al. (2018c) decompose predicted queries into different SQL clauses such as SELECT, WHERE, GROUP BY, and ORDER BY and compute scores for each clause using set matching separately. We report the following two metrics: question match, the exact set matching score over all questions, and interaction match, the exact set matching score over all interactions. The exact set matching score is 1 for each question only if all predicted SQL clauses are correct, and 1 for each interaction only if there is an exact set match for every question in the interaction.

Citation

@misc{yu2019sparc,
  title={SParC: Cross-Domain Semantic Parsing in Context},
  author={Yu et al. (2019)},
  year={2019},
  note={arXiv:1906.02285}
}

• arXiv: 1906.02285