Bird Python Eval

bird-python-eval

Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity — Hu et al. (2026) (arXiv:2601.15728, 2026)

What this evaluates

Evaluates an LLM's ability to generate executable Python code for file-based data retrieval tasks from natural language questions. It probes the model's capacity to handle explicit procedural logic, resolve ambiguous user intent, and correctly apply domain knowledge without relying on implicit database semantics.

Datasets

• BIRD-Python — total ?; splits: Simple (-1), Moderate (-1), Hard (-1); repo https://github.com/1050727345hu-web/Bird-Python

Metrics

• LLM-based Execution Accuracy (EX) (primary) — range: percent
  • The percentage of generated code snippets that execute successfully and produce output exactly matching the gold standard result. Evaluated by running the generated code against the provided data files and comparing the execution output to the expected answer.

Input / output format

Input: Database schema (or file structure), natural language question, and external domain knowledge. Prompts include constraints to ensure the generated code is executable and capable of returning valid results.

Output: Python code (typically using Pandas) that performs data retrieval/manipulation and prints or returns the final result.

Scoring recipe

def compute_ex(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        try:
            result = execute_code(pred, data_files=gold['data_files'])
            if results_match(result, gold['output']):
                correct += 1
        except Exception:
            pass
    return (correct / len(predictions)) * 100

Common pitfalls

• Conflating a model's failure to resolve ambiguous user intent with a lack of code generation capability.
• Using the original BIRD dataset without verification, as it contains noise that artificially deflates evaluation scores.
• Assuming SQL-specific code generation training transfers directly to Pandas-based data analysis tasks.

Evidence (verbatim from paper)

We benchmark Text-to-Python paradigm against the traditional Text-to-SQL approach, employing the LLM-based Execution Accuracy (EX) as the primary evaluation metric. Following the standard evaluation protocol of the BIRD benchmark (Li et al., [2024b]), our input prompts incorporate the database schema, the natural language question, and external knowledge.

Citation

@misc{hu2026benchmarking,
  title={Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2601.15728}
}

• arXiv: 2601.15728