Multi Turn Text To SQL Eval

multi-turn-text-to-sql-eval

Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL — Chen et al. (2021) (arXiv:2106.02282, 2021)

What this evaluates

Evaluates a model's ability to resolve multi-turn dialogue context into standalone questions and subsequently generate correct SQL queries. It measures both the quality of context resolution (utterance rewrite) and the accuracy of semantic parsing across individual questions and full dialogue interactions.

Datasets

• SParC — total ?; splits: dev (-1), test (-1)
• CoSQL — total ?; splits: dev (-1), test (-1)
• TASK — total ?; splits: (unstated)
• CANARD — total ?; splits: (unstated)

Metrics

• Question Match (primary) — range: [0, 1]
  • Percentage of individual questions where the predicted SQL exactly matches the golden SQL.
• Interaction Match (primary) — range: [0, 1]
  • Percentage of full interactions where all predicted SQL queries across the entire dialogue exactly match the golden SQL queries.
• Exact Match (EM) — range: [0, 1]
  • Exact match rate where the rewritten utterance prediction exactly equals the golden rewritten utterance.
• BLEU_n / ROUGE_n — range: [0, 1]
  • n-gram level similarity and overlap between predictions and golden texts.
• Rewrite F-score — range: [0, 1]
  • F-score calculated on the collection of n-grams that contain at least one word from the dialogue context.

Input / output format

Input: Rewrite task: preceding question(s) and current question. Parsing task: rewritten utterance (or original multi-turn dialogue) and database schema.

Output: Rewrite task: a single standalone rewritten question. Parsing task: a valid SQL query string.

Scoring recipe

# Question Match (QM)
qm = sum(1 for p, g in zip(pred_sqls, gold_sqls) if p == g) / len(gold_sqls)
# Interaction Match (IM)
im = sum(1 for p_turns, g_turns in zip(all_pred_sqls, all_gold_sqls) if p_turns == g_turns) / len(interactions)
# Exact Match (EM) for rewrite
em = sum(1 for p, g in zip(pred_utterances, gold_utterances) if p == g) / len(gold_utterances)

Common pitfalls

• Confusing Question Match (per-question accuracy) with Interaction Match (per-interaction accuracy), as IM is significantly stricter and drops sharply with context length.
• The official test sets for SParC and CoSQL are not publicly released, so evaluations are typically conducted on development sets or via official submission platforms.
• Annotation gaps between single-turn (Spider) and multi-turn datasets can cause performance degradation if the parser is not fine-tuned on in-domain multi-turn data.

Evidence (verbatim from paper)

Our decoupled parsing method is evaluated on two multi-turn Text-to-SQL tasks: SParC and CoSQL. Following Yu et al. ([2019b]), with Question Match and Interaction Match as the metrics. Question match means the predicted SQL equals the golden one for each question, while Interaction match indicates the predicted SQL queries of all the questions in an interaction are correct.

Citation

@misc{chen2021delta,
  title={Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2106.02282}
}

• arXiv: 2106.02282