Text To SQL Eval

text-to-sql-eval

Text-to-SQL based on Large Language Models and Database Keyword Search — Eduardo R. Nascimento et al. (arXiv:2501.13594, 2025)

What this evaluates

Evaluates a model's ability to translate natural language questions into correct SQL queries for a complex, real-world industrial database. It also probes schema-linking precision by measuring how accurately the model identifies the required tables from the schema.

Datasets

• Industrial Energy Database Benchmark — total 100; splits: test (100)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of NL questions for which the predicted SQL query returns the exact same result set (column and table values) as the ground-truth SQL query when executed on the database. Manual verification is applied to filter false positives and negatives.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for table selection in the schema linking step. Precision = |predicted_tables ∩ ground_truth_tables| / |predicted_tables|, Recall = |predicted_tables ∩ ground_truth_tables| / |ground_truth_tables|.

Input / output format

Input: Natural language question, database schema (or extended views), and optionally few-shot examples or keyword mappings.

Output: SQL query string (for compilation) or a set of table names (for schema linking).

Scoring recipe

def score_sql(pred_sql, gold_sql, db):
    pred_res = execute(pred_sql, db)
    gold_res = execute(gold_sql, db)
    return pred_res == gold_res and manual_check_false_positives(pred_res, gold_res)

def score_schema_linking(pred_tables, gold_tables):
    tp = len(pred_tables & gold_tables)
    p = tp / len(pred_tables) if pred_tables else 0
    r = tp / len(gold_tables) if gold_tables else 0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0

accuracy = sum(score_sql(p, g, db) for p, g in predictions) / len(predictions)

Common pitfalls

• Execution-based evaluation can produce false positives if semantically different SQL queries yield identical result sets; manual verification is explicitly required.
• Schema linking may return extra tables that do not break execution but can distract the LLM during query compilation, affecting accuracy independently of linking precision.
• Complexity classification (simple/medium/complex) relies on the ground-truth SQL structure, which may not perfectly align with user-perceived difficulty.

Evidence (verbatim from paper)

The experiments used an automated procedure to compare the predicted and the ground-truth SQL queries, entirely based on column and table values, and not just column and table names. ... Table [3] summarizes the results for the various alternatives. Columns under “#Correct Predicted Questions” show the number of NL questions per type correctly translated to SQL ... columns under “Accuracy” indicate the accuracy results per NL question type and the overall accuracy

Citation

@misc{nascimento2025texttosql,
  title={Text-to-SQL based on Large Language Models and Database Keyword Search},
  author={Eduardo R. Nascimento et al.},
  year={2025},
  note={arXiv:2501.13594}
}

• arXiv: 2501.13594