Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.00048 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910900196212736 |
|---|---|
| author | Hoang, Cong Duy Vu Tangari, Gioacchino Lanfranchi, Clemence Guo, Dalu Cayet, Paul Siu, Steve Dharmasiri, Don Li, Yuan-Fang Duong, Long Hilloulin, Damien Patra, Rhicheek Hong, Sungpack Chafi, Hassan |
| author_facet | Hoang, Cong Duy Vu Tangari, Gioacchino Lanfranchi, Clemence Guo, Dalu Cayet, Paul Siu, Steve Dharmasiri, Don Li, Yuan-Fang Duong, Long Hilloulin, Damien Patra, Rhicheek Hong, Sungpack Chafi, Hassan |
| contents | The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finetuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks, Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_00048 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distill-C: Enhanced NL2SQL via Distilled Customization with LLMs Hoang, Cong Duy Vu Tangari, Gioacchino Lanfranchi, Clemence Guo, Dalu Cayet, Paul Siu, Steve Dharmasiri, Don Li, Yuan-Fang Duong, Long Hilloulin, Damien Patra, Rhicheek Hong, Sungpack Chafi, Hassan Computation and Language Artificial Intelligence The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finetuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks, Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost. |
| title | Distill-C: Enhanced NL2SQL via Distilled Customization with LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.00048 |