Keyinst Eval

keyinst-eval

KeyInst: Keyword Instruction for Improving SQL Formulation in Text-to-SQL — Liu et al. (2024) (arXiv:2411.00788, 2024)

What this evaluates

This evaluation probes a model's ability to formulate correct SQL queries from natural language questions, specifically focusing on capturing structural semantics like GROUP BY, HAVING, ORDER BY, and set operations. It measures how well prompt engineering techniques or fine-tuning improve SQL generation accuracy across different database schemas and question complexities.

Datasets

• StrucQL — total ?; splits: (unstated)
• Spider — total 9034; splits: train (8000), dev (1034)
• Bird — total 12751; splits: dev (-1)

Metrics

• execution accuracy (EX) (primary) — range: [0, 1]
  • Compares the execution output of the predicted SQL query with the ground truth SQL query on the same database instance. Returns 1 if outputs match exactly, 0 otherwise, averaged over all instances.

Input / output format

Input: Natural language question, database schema, and optionally few-shot examples or keyword instructions.

Output: A single SQL query string.

Scoring recipe

def compute_ex(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        pred_result = execute_sql(pred_sql, db)
        gold_result = execute_sql(gold_sql, db)
        if pred_result == gold_result:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• KeyInst alone does not handle schema linking; combining it with SOTA schema-linking methods is required for full pipeline performance.
• Including low-priority keywords (e.g., AVG, COUNT) in the instruction can hinder LLM performance; only high-priority structural operators should be suggested.
• Execution accuracy requires identical database instances for prediction and ground truth; mismatched schemas or data versions will invalidate results.

Evidence (verbatim from paper)

Metrics We use execution accuracy (EX) to evaluate different methods. This metric compares the execution output of the predicted SQL query with that of the ground truth SQL query on same database instances.

Citation

@misc{liu2024keyinst,
  title={KeyInst: Keyword Instruction for Improving SQL Formulation in Text-to-SQL},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2411.00788}
}

• arXiv: 2411.00788