BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910066912788480 |
|---|---|
| author | Huo, Nan Xu, Xiaohan Li, Jinyang Jacobsson, Per Lin, Shipei Qin, Bowen Hui, Binyuan Li, Xiaolong Qu, Ge Si, Shuzheng Han, Linheng Alexander, Edward Zhu, Xintong Qin, Rui Yu, Ruihan Jin, Yiyao Zhou, Feige Zhong, Weihao Chen, Yun Liu, Hongyu Ma, Chenhao Ozcan, Fatma Papakonstantinou, Yannis Cheng, Reynold |
| author_facet | Huo, Nan Xu, Xiaohan Li, Jinyang Jacobsson, Per Lin, Shipei Qin, Bowen Hui, Binyuan Li, Xiaolong Qu, Ge Si, Shuzheng Han, Linheng Alexander, Edward Zhu, Xintong Qin, Rui Yu, Ruihan Jin, Yiyao Zhou, Feige Zhong, Weihao Chen, Yun Liu, Hongyu Ma, Chenhao Ozcan, Fatma Papakonstantinou, Yannis Cheng, Reynold |
| contents | Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fall short by treating conversation histories as static context or limiting evaluation to read-only operations, failing to reflect production-grade database assistant challenges. We introduce BIRD-INTERACT, a benchmark that restores this realism through: (1) a comprehensive interaction environment coupling each database with a hierarchical knowledge base, metadata files, and a function-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from errors without human supervision; (2) two evaluation settings consisting of a pre-defined conversational protocol (c-Interact) and an open-ended agentic setting (a-Interact) where models autonomously decide when to query the user simulator or explore the environment; (3) a challenging task suite covering the full CRUD spectrum for business-intelligence and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600 tasks, up to 11,796 interactions) for comprehensive performance assessment, and BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed behavioral analysis and rapid method development. Our empirical results highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in c-Interact and 17.00% in a-Interact. Analysis via memory grafting and Interaction Test-time Scaling validates the importance of effective interaction for complex, dynamic text-to-SQL tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05318 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions Huo, Nan Xu, Xiaohan Li, Jinyang Jacobsson, Per Lin, Shipei Qin, Bowen Hui, Binyuan Li, Xiaolong Qu, Ge Si, Shuzheng Han, Linheng Alexander, Edward Zhu, Xintong Qin, Rui Yu, Ruihan Jin, Yiyao Zhou, Feige Zhong, Weihao Chen, Yun Liu, Hongyu Ma, Chenhao Ozcan, Fatma Papakonstantinou, Yannis Cheng, Reynold Artificial Intelligence Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fall short by treating conversation histories as static context or limiting evaluation to read-only operations, failing to reflect production-grade database assistant challenges. We introduce BIRD-INTERACT, a benchmark that restores this realism through: (1) a comprehensive interaction environment coupling each database with a hierarchical knowledge base, metadata files, and a function-driven user simulator, enabling models to solicit clarifications, retrieve knowledge, and recover from errors without human supervision; (2) two evaluation settings consisting of a pre-defined conversational protocol (c-Interact) and an open-ended agentic setting (a-Interact) where models autonomously decide when to query the user simulator or explore the environment; (3) a challenging task suite covering the full CRUD spectrum for business-intelligence and operational use cases, guarded by executable test cases. Each task features ambiguous and follow-up sub-tasks requiring dynamic interaction. The suite comprises BIRD-INTERACT-FULL (600 tasks, up to 11,796 interactions) for comprehensive performance assessment, and BIRD-INTERACT-LITE (300 tasks with simplified databases) for detailed behavioral analysis and rapid method development. Our empirical results highlight BIRD-INTERACT's difficulty: GPT-5 completes only 8.67% of tasks in c-Interact and 17.00% in a-Interact. Analysis via memory grafting and Interaction Test-time Scaling validates the importance of effective interaction for complex, dynamic text-to-SQL tasks. |
| title | BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.05318 |