Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.03170 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916277035991040 |
|---|---|
| author | Nooralahzadeh, Farhad Zhang, Yi Smith, Ellery Maennel, Sabine Matthey-Doret, Cyril de Fondville, Raphaël Stockinger, Kurt |
| author_facet | Nooralahzadeh, Farhad Zhang, Yi Smith, Ellery Maennel, Sabine Matthey-Doret, Cyril de Fondville, Raphaël Stockinger, Kurt |
| contents | The potential for improvements brought by Large Language Models (LLMs) in Text-to-SQL systems is mostly assessed on monolingual English datasets. However, LLMs' performance for other languages remains vastly unexplored. In this work, we release the StatBot.Swiss dataset, the first bilingual benchmark for evaluating Text-to-SQL systems based on real-world applications. The StatBot.Swiss dataset contains 455 natural language/SQL-pairs over 35 big databases with varying level of complexity for both English and German.
We evaluate the performance of state-of-the-art LLMs such as GPT-3.5-Turbo and mixtral-8x7b-instruct for the Text-to-SQL translation task using an in-context learning approach. Our experimental analysis illustrates that current LLMs struggle to generalize well in generating SQL queries on our novel bilingual dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_03170 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StatBot.Swiss: Bilingual Open Data Exploration in Natural Language Nooralahzadeh, Farhad Zhang, Yi Smith, Ellery Maennel, Sabine Matthey-Doret, Cyril de Fondville, Raphaël Stockinger, Kurt Computation and Language The potential for improvements brought by Large Language Models (LLMs) in Text-to-SQL systems is mostly assessed on monolingual English datasets. However, LLMs' performance for other languages remains vastly unexplored. In this work, we release the StatBot.Swiss dataset, the first bilingual benchmark for evaluating Text-to-SQL systems based on real-world applications. The StatBot.Swiss dataset contains 455 natural language/SQL-pairs over 35 big databases with varying level of complexity for both English and German. We evaluate the performance of state-of-the-art LLMs such as GPT-3.5-Turbo and mixtral-8x7b-instruct for the Text-to-SQL translation task using an in-context learning approach. Our experimental analysis illustrates that current LLMs struggle to generalize well in generating SQL queries on our novel bilingual dataset. |
| title | StatBot.Swiss: Bilingual Open Data Exploration in Natural Language |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.03170 |