TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917033144221696 |
|---|---|
| author | Harrasse, Abir Quirke, Philip Neo, Clement Nathawani, Dhruv Marks, Luke Abdullah, Amir |
| author_facet | Harrasse, Abir Quirke, Philip Neo, Clement Nathawani, Dhruv Marks, Luke Abdullah, Amir |
| contents | Mechanistic interpretability research faces a gap between analyzing simple circuits in toy tasks and discovering features in large models. To bridge this gap, we propose text-to-SQL generation as an ideal task to study, as it combines the formal structure of toy tasks with real-world complexity. We introduce TinySQL, a synthetic dataset, progressing from basic to advanced SQL operations, and train models ranging from 33M to 1B parameters to establish a comprehensive testbed for interpretability. We apply multiple complementary interpretability techniques, including Edge Attribution Patching and Sparse Autoencoders, to identify minimal circuits and components supporting SQL generation. We compare circuits for different SQL subskills, evaluating their minimality, reliability, and identifiability. Finally, we conduct a layerwise logit lens analysis to reveal how models compose SQL queries across layers: from intent recognition to schema resolution to structured generation. Our work provides a robust framework for probing and comparing interpretability methods in a structured, progressively complex setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12730 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research Harrasse, Abir Quirke, Philip Neo, Clement Nathawani, Dhruv Marks, Luke Abdullah, Amir Machine Learning Artificial Intelligence Databases Mechanistic interpretability research faces a gap between analyzing simple circuits in toy tasks and discovering features in large models. To bridge this gap, we propose text-to-SQL generation as an ideal task to study, as it combines the formal structure of toy tasks with real-world complexity. We introduce TinySQL, a synthetic dataset, progressing from basic to advanced SQL operations, and train models ranging from 33M to 1B parameters to establish a comprehensive testbed for interpretability. We apply multiple complementary interpretability techniques, including Edge Attribution Patching and Sparse Autoencoders, to identify minimal circuits and components supporting SQL generation. We compare circuits for different SQL subskills, evaluating their minimality, reliability, and identifiability. Finally, we conduct a layerwise logit lens analysis to reveal how models compose SQL queries across layers: from intent recognition to schema resolution to structured generation. Our work provides a robust framework for probing and comparing interpretability methods in a structured, progressively complex setting. |
| title | TinySQL: A Progressive Text-to-SQL Dataset for Mechanistic Interpretability Research |
| topic | Machine Learning Artificial Intelligence Databases |
| url | https://arxiv.org/abs/2503.12730 |