Beaver Eval

beaver-eval

BEAVER: An Enterprise Benchmark for Text-to-SQL — Chen et al. (2024) (arXiv:2409.02038, 2024)

What this evaluates

Probes LLMs' ability to generate correct SQL queries from natural language questions over complex, enterprise-scale databases. It specifically evaluates handling of high schema complexity, multi-table joins, aggregations, and column-to-table mapping in real-world business contexts.

Datasets

• BEAVER — total 203; splits: test (203)

Metrics

• execution accuracy (primary) — range: [0, 1]
  • Percentage of generated SQL statements that produce the exact same result set as the gold SQL when executed against the provided database instances.

Input / output format

Input: A natural language question paired with a database schema (table names, column names, data types) and table instances (row data).

Output: A single SQL statement.

Scoring recipe

def score(predictions, golds, databases):
    correct = 0
    for pred_sql, gold_sql, db in zip(predictions, golds, databases):
        try:
            pred_result = db.execute(pred_sql)
            gold_result = db.execute(gold_sql)
            if set(pred_result) == set(gold_result):
                correct += 1
        except Exception:
            pass
    return correct / len(predictions)

Common pitfalls

• High schema complexity makes column and instance mapping challenging for models.
• Instance mapping is explicitly not annotated in the dataset due to considerable complexity.
• Models trained on public benchmarks often fail to generalize due to missing enterprise-scale multi-table joins and business-domain complexity.

Evidence (verbatim from paper)

Following the standard problem setup of text-to-SQL, the input to an LLM includes a natural language question and a database of tables, and the output is a SQL statement whose execution answers the user’s question.

Citation

@misc{chen2024beaver,
  title={BEAVER: An Enterprise Benchmark for Text-to-SQL},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2409.02038}
}

• arXiv: 2409.02038