Dab Data Agent Benchmark

Evaluate data agents on realistic multi-database queries across 54 tasks spanning 12 datasets, 9 domains, and 4 DBMS systems. Reveals frontier models achieve only 38% pass@1 accuracy, with 85% of failures from incorrect planning rather than data selection. Benchmark captures critical challenges: heterogeneous database integration, identifier reconciliation, unstructured text extraction, and domain knowledge. Identifies optimal exploration ratio (~20% of operations) and cost-efficiency differences across models.

adu2021 Updated 7 repo stars

File contents

adu2021/skillxiv commit aa2fa31d8c

Frequently asked questions

npx get-skill add adu2021/dab-data-agent-benchmark