S3eval SQL Execution Eval

s3eval-sql-execution-eval

S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models — Lei et al. (2023) (arXiv:2310.15147, 2023)

What this evaluates

Evaluates large language models' ability to perform complex, multi-step reasoning and data dependency tracking by executing SQL queries over synthetic, arbitrarily long tables. It probes contextual reasoning, long-context understanding, and structured data manipulation capabilities beyond traditional benchmarks.

Datasets

• S3Eval — total ?; splits: Easy (-1), General (-1); repo https://github.com/lfy79001/S3Eval

Metrics

• SQL execution performance (primary) — range: percent
  • Percentage of queries where the model's predicted output exactly matches the ground truth execution result. Computed as (correct predictions / total queries) * 100.

Input / output format

Input: A synthetic table (with arbitrary length and structure) provided in text format, followed by a SQL query to be executed on that table.

Output: The exact result of the SQL execution (e.g., a list of values or a single scalar), typically returned in plain text or markdown format.

Scoring recipe

def compute_performance(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if normalize(pred) == normalize(gold):
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Models may hallucinate table structures or column names not present in the prompt.
• Execution performance requires exact string/format matching; minor formatting differences (e.g., extra whitespace, different list syntax) can cause false negatives.
• Long-context settings (20K-80K tokens) often cause performance dips due to context window limits or attention degradation, not just reasoning failure.

Evidence (verbatim from paper)

Table 4: SQL Execution Task Performance on different LLMs. SQL execution is a suitable task because SQL can be intertranslated with an natural question. ChatGPT's performance (markdown) improves from 38.0 to 48.5 when using chain-of-thought prompts.

Citation

@misc{lei2023s3eval,
  title={S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models},
  author={Lei et al. (2023)},
  year={2023},
  note={arXiv:2310.15147}
}

• arXiv: 2310.15147