An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908873032466432 |
|---|---|
| author | Pham, Trinh Nguyen, Thanh Tam Huynh, Viet Yin, Hongzhi Nguyen, Quoc Viet Hung |
| author_facet | Pham, Trinh Nguyen, Thanh Tam Huynh, Viet Yin, Hongzhi Nguyen, Quoc Viet Hung |
| contents | Recent advances in large language models has strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07841 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data Pham, Trinh Nguyen, Thanh Tam Huynh, Viet Yin, Hongzhi Nguyen, Quoc Viet Hung Computation and Language Recent advances in large language models has strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL. |
| title | An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.07841 |