DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929388091604992 |
|---|---|
| author | Zhou, Fan Xue, Siqiao Qi, Danrui Shi, Wenhui Zhao, Wang Wei, Ganglin Zhang, Hongyang Jiang, Caigai Jiang, Gangwei Chu, Zhixuan Chen, Faqiang |
| author_facet | Zhou, Fan Xue, Siqiao Qi, Danrui Shi, Wenhui Zhao, Wang Wei, Ganglin Zhang, Hongyang Jiang, Caigai Jiang, Gangwei Chu, Zhixuan Chen, Faqiang |
| contents | Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11434 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models Zhou, Fan Xue, Siqiao Qi, Danrui Shi, Wenhui Zhao, Wang Wei, Ganglin Zhang, Hongyang Jiang, Caigai Jiang, Gangwei Chu, Zhixuan Chen, Faqiang Databases Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub. |
| title | DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models |
| topic | Databases |
| url | https://arxiv.org/abs/2406.11434 |