SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910058324951040 |
|---|---|
| author | Zhang, Bin Ye, Yuxiao Du, Guoqing Hu, Xiaoru Li, Zhishuai Liu, Chi Harold Xu, Zhiwei Fan, Guoliang Zhao, Rui Li, Ziyue Mao, Hangyu |
| author_facet | Zhang, Bin Ye, Yuxiao Du, Guoqing Hu, Xiaoru Li, Zhishuai Liu, Chi Harold Xu, Zhiwei Fan, Guoliang Zhao, Rui Li, Ziyue Mao, Hangyu |
| contents | Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods.Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions. To address the aforementioned issues, we firstly construct a new dataset designed to mitigate the risk of overfitting in LLMs. Then we formulate five evaluation tasks to comprehensively assess the performance of diverse methods across various LLMs throughout the Text-to-SQL process.Our study highlights the performance disparities among LLMs and proposes optimal in-context learning solutions tailored to each task. These findings offer valuable insights for facilitating the development of LLM-based Text-to-SQL systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_02951 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models Zhang, Bin Ye, Yuxiao Du, Guoqing Hu, Xiaoru Li, Zhishuai Liu, Chi Harold Xu, Zhiwei Fan, Guoliang Zhao, Rui Li, Ziyue Mao, Hangyu Computation and Language Artificial Intelligence Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods.Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions. To address the aforementioned issues, we firstly construct a new dataset designed to mitigate the risk of overfitting in LLMs. Then we formulate five evaluation tasks to comprehensively assess the performance of diverse methods across various LLMs throughout the Text-to-SQL process.Our study highlights the performance disparities among LLMs and proposes optimal in-context learning solutions tailored to each task. These findings offer valuable insights for facilitating the development of LLM-based Text-to-SQL systems. |
| title | SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.02951 |