Spider 2.0 Eval

spider-2.0-eval

Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows — Lei et al. (2024) (arXiv:2411.07763, 2024)

What this evaluates

Evaluates language models' ability to perform real-world enterprise text-to-SQL workflows. It probes agentic reasoning, multi-step data transformation, database schema navigation, and SQL dialect adaptation across complex, long-context tasks.

Datasets

• Spider 2.0 — total 632; splits: test (-1)
• Spider 2.0-lite — total ?; splits: test (-1)
• Spider 2.0-snow — total ?; splits: test (-1)

Metrics

• Success Rate (SR) (primary) — range: percent
  • Proportion of task instances successfully completed. Calculated as the average of binary scores (0 or 1) from the evaluation script across all instances.
• Execution Accuracy (EX) — range: percent
  • Proportion of instances where the generated SQL query executes successfully and matches the gold answer. Computed by running the generated SQL against the database and comparing results.

Input / output format

Input: Natural language instruction/question, database schema, codebase, metadata, and documentation context.

Output: SQL query, result table, or database state.

Scoring recipe

def compute_sr(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        score = run_evaluation_script(pred, gold)  # returns 0 or 1
        scores.append(score)
    return sum(scores) / len(scores)

def compute_ex(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        score = check_execution_match(pred, gold)  # returns 0 or 1
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• In table-based evaluations, predicted results may contain numerous columns not in the gold answer because questions do not explicitly specify which columns to return. Evaluation scripts ignore irrelevant columns and focus only on essential components.
• Difficulty levels are assigned based on whitespace token counts of gold SQL (<80 Easy, 80-159 Medium, ≥160 Hard), which may not fully capture semantic or structural complexity.
• Models must handle multi-step agentic reasoning and SQL dialect differences (e.g., BigQuery vs. Snowflake), not just single-shot query generation.

Evidence (verbatim from paper)

For Spider 2.0, we use the Success Rate (SR) metric, which measures the proportion of task instances successfully completed. For Spider 2.0-lite and Spider 2.0-snow, the output for each task must be an SQL, we use the widely used metric Execution Accuracy (EX)(Yu et al., 2018; Li et al., 2024b). We employ the execution-based focused evaluation (App.A) to determine the success of each result for Spider 2.0 and assess the accuracy of SQL execution results for Spider 2.0-lite. The evaluation scripts are designed to accept output in the form of strings, tables, or database. For each example, an evaluation script is run for each example, producing a score of either 0 or 1.

Citation

@misc{lei2024spider2.0,
  title={Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows},
  author={Lei et al. (2024)},
  year={2024},
  note={arXiv:2411.07763}
}

• arXiv: 2411.07763