Bird Critic Eval

bird-critic-eval

SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications — Li et al. (2025) (arXiv:2506.18951, 2025)

What this evaluates

This benchmark evaluates an LLM's ability to debug and fix real-world, user-reported SQL issues. It probes the model's capacity to identify logical, syntactic, and semantic flaws in existing queries and generate correct, functional SQL replacements across different database dialects and complexity levels.

Datasets

• BIRD-CRITIC — total ?; splits: BIRD-CRITIC-PG (-1), BIRD-CRITIC-Multi (-1)

Metrics

• Success Rate (SR) (primary) — range: percent
  • The percentage of test cases where the model successfully resolves the reported SQL issue and produces a correct, functional query. Calculated as (number of correctly resolved issues / total number of issues) * 100.

Input / output format

Input: A user-reported SQL issue description accompanied by a buggy or incorrect SQL query that needs debugging.

Output: A corrected SQL query that resolves the reported issue.

Scoring recipe

def calculate_sr(predictions, gold):
    correct = 0
    for pred, gold_sql in zip(predictions, gold):
        if execute_and_match(pred, gold_sql): # Semantic/functional equivalence check
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Assuming the task is standard SQL generation from natural language; it specifically requires debugging existing buggy queries based on user-reported issues.
• Overlooking dialect-specific syntax variations when evaluating the multi-dialect split, as models trained primarily on one dialect may fail on others.
• Expecting high baseline performance; the benchmark reveals that even state-of-the-art reasoning models achieve only ~33-39% success rates, indicating significant difficulty.

Evidence (verbatim from paper)

These models surpass general-purpose counterparts by an average Success Rate (SR) of 6.13 % on PostgreSQL issues and 8.03 % on multi-dialect issues.

Citation

@misc{li2025swe_sql,
  title={SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2506.18951}
}

• arXiv: 2506.18951