Tragesql Eval

tragesql-eval

Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL — Zhang et al. (2020) (arXiv:2010.12634, 2020)

What this evaluates

This benchmark evaluates a model's ability to classify the intention of a natural language question relative to a database schema. It probes whether the model can distinguish between answerable queries, questions requiring external knowledge, ambiguous queries, grammatically invalid non-SQL questions, and questions unrelated to the schema.

Datasets

• TRIAGESQL — total ?; splits: train (384859), dev (86409), test (2500); repo https://github.com/chatc/TriageSQL

Metrics

• Macro F1 (primary) — range: [0, 1]
  • The unweighted mean of the F1 scores computed for each of the five intention classes (Improper, ExtKnow, Ambiguous, Non-SQL, Answerable). F1 for a class is the harmonic mean of its precision and recall.

Input / output format

Input: A pair consisting of a natural language question and a database schema, with a special token separating the question from each schema column.

Output: A single class label from the set: {Improper, ExtKnow, Ambiguous, Non-SQL, Answerable}.

Scoring recipe

def macro_f1(y_true, y_pred, classes):
    f1_scores = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Models frequently confuse 'ExtKnow' (external knowledge required) with 'Answerable' questions, especially when the schema contains non-mentioned columns that could theoretically answer the query.
• Ambiguous questions receive very low F1 scores due to inherent difficulty in distinguishing schema/value ambiguity from other classes.
• The dataset is highly imbalanced across classes, so accuracy is misleading; Macro F1 must be used to evaluate performance uniformly across all intention types.

Evidence (verbatim from paper)

Table 4 shows the result of the RoBERTa model on the proposed test set, achieving a 60% F1 score on average. Some question types can be classified with high F1 scores, such as Improper questions and Non-SQL questions unanswerable by common SQL grammar. However, it only obtains 17% F1 score on Ambiguous questions.

Citation

@misc{zhang2020tragesql,
  title={Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2010.12634}
}

• arXiv: 2010.12634