Tag Plus Eval

tag-plus-eval

SEMA-SQL: Beyond Traditional Relational Querying with Large Language Models — Yin Lin et al. (2026) (arXiv:2604.23477, 2026)

What this evaluates

This benchmark evaluates an LLM's ability to generate and execute hybrid relational queries that combine traditional SQL operations with semantic reasoning over textual data. It probes capabilities like semantic joins, information extraction, and multi-hop reasoning by measuring execution accuracy against expert-verified ground truth.

Datasets

• TAG+ — total 120; splits: test (120); repo https://github.com/semasql/SEMA-SQL

Metrics

• execution accuracy (primary) — range: percent
  • Percentage of queries where the system's output matches the expert-labeled ground truth. For subjective tasks (e.g., summarization, ranking), an LLM judge (GPT-5) verifies if outputs capture key information or demonstrate sound reasoning.

Input / output format

Input: Natural language questions paired with relational database schemas and data (including textual columns requiring world knowledge or semantic reasoning).

Output: Executable Hybrid Relational Algebra (HRA) or SQL queries, followed by the final query results/answers.

Scoring recipe

correct = 0
for q, gt in dataset:
    res = model.generate_and_execute(q)
    if res == gt:
        correct += 1
    elif is_subjective(q):
        if llm_judge('GPT-5').verify(res, gt):
            correct += 1
return (correct / len(dataset)) * 100

Common pitfalls

• Subjective tasks rely on GPT-5 as an LLM judge, which may introduce evaluation bias or inconsistency compared to exact match.
• Baseline systems like BlendSQL batch multiple entries per LLM call to save tokens, which artificially lowers accuracy but improves token efficiency, making direct accuracy comparisons misleading without context.
• Row-wise data imputation baselines (e.g., HQDL) frequently timeout on large relations, skewing average token/latency metrics if timeouts are not handled consistently.

Evidence (verbatim from paper)

Table 3 presents the execution accuracy and token usage of all systems, where we interact with each system via its natural language or code interface as summarized in Table 2. For TAG+ questions, we consider results identical to ground truth as correct. For subjective tasks (e.g., summarizing textual comments or ranking schools by perceived promise), we use GPT-5 as an LLM judge to assess whether summarization outputs capture key information effectively, and whether ranking results demonstrate sound reasoning.

Citation

@misc{lin2026semasql,
  title={SEMA-SQL: Beyond Traditional Relational Querying with Large Language Models},
  author={Yin Lin et al. (2026)},
  year={2026},
  note={arXiv:2604.23477}
}

• arXiv: 2604.23477