Text To SQL Annotation Error Eval

text-to-sql-annotation-error-eval

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards — Jin et al. (2026) (arXiv:2601.08778, 2026)

What this evaluates

Evaluates the reliability of text-to-SQL benchmarks by quantifying annotation error rates and measuring how these errors distort agent execution accuracy and leaderboard rankings.

Datasets

• BIRD — total ?; splits: dev (-1)
• Spider 2.0-Snow — total ?; splits: full (-1)

Metrics

• annotation error rate (primary) — range: percent
  • Calculated as (number of identified annotation errors / total evaluated examples) × 100. Errors are classified as semantic mismatches, domain knowledge gaps, or ambiguity in the gold SQL annotation.
• execution accuracy — range: [0, 1]
  • Standard text-to-SQL metric measuring the fraction of examples where the model's generated SQL produces the exact same result set as the gold SQL when executed against the database.

Input / output format

Input: Natural language question, database schema/context, and (for error detection) the gold SQL annotation.

Output: SQL query.

Scoring recipe

def compute_annotation_error_rate(errors, total):
    return (errors / total) * 100

def compute_execution_accuracy(predictions, golds, db_executor):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if db_executor.run(pred) == db_executor.run(gold):
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Relying on unverified gold annotations without human-in-the-loop auditing, which masks systematic benchmark corruption
• Using hidden test sets for local evaluation instead of the designated development set, violating benchmark protocols
• Ignoring domain-specific SQL syntax variations that cause false negatives in execution matching

Evidence (verbatim from paper)

In Section[5], we sample 100 examples from the BIRD Dev set and manually correct annotation errors by applying SAPAR. To answer Q2, we then re-evaluate all 16 open-source agents listed on the BIRD leaderboard on both the original and the corrected versions of this sampled subset. We quantify changes in execution accuracy and the relative ranking among these agents.

Citation

@misc{jin2026pervasive,
  title={Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards},
  author={Jin et al. (2026)},
  year={2026},
  note={arXiv:2601.08778}
}

• arXiv: 2601.08778