Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911685256675328 |
|---|---|
| author | Gigant, Théo Peng, Bowen Quesnelle, Jeffrey |
| author_facet | Gigant, Théo Peng, Bowen Quesnelle, Jeffrey |
| contents | Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_27263 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation Gigant, Théo Peng, Bowen Quesnelle, Jeffrey Computation and Language Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases. |
| title | Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.27263 |