Semanticagent Eval

semanticagent-eval

SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesis — Gao et al. (2026) (arXiv:2604.21414, 2026)

What this evaluates

Evaluates text-to-SQL generation capabilities across cross-domain parsing, knowledge-intensive reasoning, and enterprise-level SQL workflows. It also assesses the semantic validity, execution correctness, and diversity of synthetically generated training data.

Datasets

• Spider — total 10181; splits: dev (-1), test (-1)
• BIRD — total 12751; splits: dev (-1)
• Spider2.0 — total 135; splits: test (135)
• EHRSQL — total ?; splits: (unstated)
• ScienceBenchmark — total ?; splits: (unstated)
• Spider-Syn — total ?; splits: (unstated)
• Spider-Realistic — total ?; splits: (unstated)
• Spider-DK — total ?; splits: (unstated)

Metrics

• test-suite accuracy (TS) (primary) — range: [0, 1]
  • Fraction of predictions that match the gold execution results across all provided test databases for a given schema.
• execution accuracy (EX) (primary) — range: [0, 1]
  • Fraction of predictions whose execution results match the gold execution result on a single database.
• Semantic Alignment (SA) — range: [0, 1]
  • LLM-based consistency score measuring semantic alignment between the natural language question and the generated SQL query.
• Successful Execution Rate (SER) — range: [0, 1]
  • Proportion of synthesized SQL queries that execute without syntax or runtime errors.

Input / output format

Input: Natural language question paired with database schema information (table names, column names, and sampled cell values).

Output: A single SQL query string.

Scoring recipe

def compute_accuracy(predictions, golds, db):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if execute_sql(pred, db) == execute_sql(gold, db):
            correct += 1
    return correct / len(predictions)
# For TS, iterate over multiple test databases per schema and require match on all.

Common pitfalls

• Synthetic data contamination: generated examples may accidentally overlap with public evaluation splits; requires n-gram filtering against the eval set.
• Execution vs. semantic correctness: standard EX/TS metrics only verify syntactic/execution equivalence, not whether the SQL actually answers the natural language question semantically.
• Schema-only generation constraint: synthetic data is built exclusively from schema structures and sampled cells without using gold question-SQL pairs, altering the data distribution compared to standard fine-tuning.

Evidence (verbatim from paper)

Spider (dev), Spider-Syn, and Spider-Realistic are evaluated with test-suite accuracy (TS), while the remaining benchmarks are evaluated with execution accuracy (EX). EX compares execution results on a single database, whereas TS evaluates predictions over multiple test databases.

Citation

@misc{gao2026semanticagent,
  title={SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesis},
  author={Gao et al. (2026)},
  year={2026},
  note={arXiv:2604.21414}
}

• arXiv: 2604.21414