| name | dab-data-agent-benchmark |
| title | DAB: Data Agent Benchmark |
| version | 0.0.3 |
| engine | skillxiv-v0.0.3-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2603.20576 |
| keywords | ["Data Agent","Multi-Database Querying","Benchmark Design","Agent Evaluation","Structured Reasoning"] |
| description | Evaluate data agents on realistic multi-database queries across 54 tasks spanning 12 datasets, 9 domains, and 4 DBMS systems. Reveals frontier models achieve only 38% pass@1 accuracy, with 85% of failures from incorrect planning rather than data selection. Benchmark captures critical challenges: heterogeneous database integration, identifier reconciliation, unstructured text extraction, and domain knowledge. Identifies optimal exploration ratio (~20% of operations) and cost-efficiency differences across models. |
| category | Evaluation Infrastructure |
Benchmark Scope & Design
DAB evaluates data agents on realistic query tasks that require multi-step reasoning across heterogeneous structured data sources. The benchmark comprises 54 queries distributed across:
- 12 Diverse Datasets: Patents, academic publications, financial records, product catalogs, public health databases, geographic data
- 9 Domains: Healthcare, finance, technology, e-commerce, law, science, demographics, infrastructure, government
- 4 Database Systems: SQL (PostgreSQL, MySQL), NoSQL (MongoDB), spreadsheet formats, requiring agents to handle different schema interfaces
Query complexity ranges from single-table lookups to multi-stage joins across incompatible databases with data cleaning steps. This heterogeneity ensures benchmarking real-world agent challenges rather than uniform database navigation.
Task Definition & Evaluation Metrics
Query Format: Natural language instruction plus optional context. Example: "Find patents filed by XYZ company in biotechnology sector in the last 5 years, ranked by citation count." Agent must interpret intent, plan exploration, locate relevant tables, construct queries, and extract/aggregate results.
Evaluation Criteria:
- pass@1: First-attempt correctness; best model achieves only 38% accuracy
- pass@50: Accuracy with up to 50 retry attempts; performance caps at 69%, indicating fundamental limitations not just sampling variance
- Unsolved problems: Patents dataset remains 0% solvable across all models, suggesting domain-specific knowledge barriers
Success Definition: Returned data matches ground-truth answer set exactly. Partial matches or near-correct queries count as failures, reflecting real-world requirements where approximate answers are unacceptable.
Error Analysis: Root Causes
The benchmark reveals systematic failure modes across all frontier models:
Planning Failures (85% of errors): Agents construct incorrect query plans even when data is discoverable. Mistakes cluster in:
- Multi-database integration: Failing to identify which tables store required data when information is scattered across systems
- Join strategy errors: Using incorrect join columns or join types when identifiers don't match cleanly
- Aggregation mistakes: Incorrect GROUP BY clauses, SUM vs COUNT confusion, or wrong filtering order
Data Selection Failures (15% of errors): Correctly planned queries access wrong data sources or miss relevant tables entirely.
