MIT and partners release BEAVER, a large-scale benchmark dataset for evaluating how well AI models can turn natural language questions into SQL database queries using real enterprise data.

What happened

Researchers from MIT, Harvard University, TU Munich, Greenshoe Inc., Intel, and AWS AI Labs created BEAVER, a dataset containing 9128 queries spanning 812 tables across 19 diverse domains. Of these, 7978 queries are publicly released, while the remaining portion is held out as a private test set. The dataset includes annotations for five subtasks: multi-table retrieval, join key detection, column mapping, domain knowledge extraction, and query decomposition.

What to watch

The dataset distinguishes between three categories of queries: complex queries without domain knowledge, domain-specific queries with minimal complexity, and domain-specific complex queries. This structure allows evaluators to measure performance across different levels of difficulty and business relevance. The 7978 public queries are available for research, while the private test set remains for independent evaluation.