Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915702273736704 |
|---|---|
| author | Cromp, Sonia GNVV, Satya Sai Srinath Namburi Alkhudhayri, Mohammed Cao, Catherine Guo, Samuel Roberts, Nicholas Sala, Frederic |
| author_facet | Cromp, Sonia GNVV, Satya Sai Srinath Namburi Alkhudhayri, Mohammed Cao, Catherine Guo, Samuel Roberts, Nicholas Sala, Frederic |
| contents | While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02152 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis Cromp, Sonia GNVV, Satya Sai Srinath Namburi Alkhudhayri, Mohammed Cao, Catherine Guo, Samuel Roberts, Nicholas Sala, Frederic Machine Learning Computation and Language I.2.6 While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well. |
| title | Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis |
| topic | Machine Learning Computation and Language I.2.6 |
| url | https://arxiv.org/abs/2503.02152 |