Cosql Eval

cosql-eval

CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases — Tao Yu et al. (2019) (arXiv:1909.05378, 2019)

What this evaluates

Evaluates conversational text-to-SQL systems on cross-domain database querying. It probes dialogue state tracking via SQL grounding, response generation from query results, and user intent/dialogue act prediction under real-world ambiguity and clarification dynamics.

Datasets

• CoSQL — total 30000; splits: dev (-1), test (-1)

Metrics

• Question Match (primary) — range: [0, 1]
  • Exact match accuracy: 1 if predicted SQL equals gold SQL, else 0. Averaged over all questions.
• Interaction Match — range: [0, 1]
  • Exact match accuracy over a full dialogue: 1 if all predicted SQLs in the interaction match gold SQLs, else 0. Averaged over all interactions.
• BLEU — range: percent
  • Standard BLEU score measuring n-gram overlap between generated natural language descriptions and reference texts.
• Logic Correctness Rate (LCR) — range: [0, 1]
  • Human evaluation where raters assign 0 or 1 based on whether the generated description correctly captures the SQL logic. Final score uses majority vote across raters.
• Grammar — range: [1, 5]
  • Human evaluation on a 1-5 scale assessing grammatical correctness of the generated description. Average score reported.
• User Dialog Act Accuracy — range: [0, 1]
  • Exact match accuracy for predicting the user's dialogue act label (e.g., INFORM_SQL, AMBIGUOUS, THANK_YOU).

Input / output format

Input: Conversational natural language turns in context of a database schema, with prior dialogue history.

Output: For DST: executable SQL queries. For response generation: natural language descriptions of SQL results. For dialog act prediction: categorical dialogue act labels.

Scoring recipe

def score_cosql(preds, golds, human_raters=None):
    q_match = sum(1 for p, g in zip(preds['sql'], golds['sql']) if p == g) / len(golds['sql'])
    i_match = sum(1 for p, g in zip(preds['interactions'], golds['interactions']) if p == g) / len(golds['interactions'])
    bleu = compute_bleu(preds['text'], golds['text'])
    if human_raters:
        lcr_scores, gram_scores = [], []
        for p, g in zip(preds['text'], golds['text']):
            votes_lcr = [r.score_logic(p, g) for r in human_raters]  # 0/1
            votes_gram = [r.score_grammar(p) for r in human_raters]  # 1-5
            lcr_scores.append(1 if sum(votes_lcr) > len(votes_lcr)/2 else 0)
            gram_scores.append(sum(votes_gram)/len(votes_gram))
        lcr = sum(lcr_scores)/len(lcr_scores)
        grammar = sum(gram_scores)/len(gram_scores)
    return {'question_match': q_match, 'interaction_match': i_match, 'bleu': bleu, 'lcr': lcr, 'grammar': grammar}

Common pitfalls

• CoSQL DST is significantly harder than SParC due to crowd-sourced users, ambiguous questions, and complex intent switches.
• BLEU scores often misalign with human evaluations (LCR/Grammar), as neural models may generate grammatically fluent but logically incorrect SQL descriptions.
• High accuracy on simple dialogue acts (e.g., THANK_YOU) masks poor performance on complex acts like INFER_SQL and AMBIGUOUS (F-scores ~10%).

Evidence (verbatim from paper)

We use the same evaluation metrics used by the SParC dataset (Yu et al., 2019) to evaluate the model's performance on all questions and interactions (dialogs). ... To compute LCR and grammar score, we randomly sampled 100 descriptions generated by each model. Three students proficient in English participated in the evaluation, They were asked to choose a score 0 or 1 for LCR, and 1 to 5 for grammar check (the larger, the better). For LCR, the final score was decided by majority vote.

Citation

@misc{yu2019cosql,
  title={CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases},
  author={Tao Yu et al. (2019)},
  year={2019},
  note={arXiv:1909.05378}
}

• arXiv: 1909.05378