Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2512.20757 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911335929872384 |
|---|---|
| author | Altıntaş, Gül Sena Ehghaghi, Malikeh Lester, Brian Liu, Fengyuan Zhao, Wanru Ciccone, Marco Raffel, Colin |
| author_facet | Altıntaş, Gül Sena Ehghaghi, Malikeh Lester, Brian Liu, Fengyuan Zhao, Wanru Ciccone, Marco Raffel, Colin |
| contents | Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we train fourteen models that use different tokenizers but are otherwise identical using the same architecture, dataset, training budget, and initialization. Additionally, we curate and release a new benchmark that specifically measures model performance subject to real-world perturbations that are likely to influence tokenization. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_20757 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior Altıntaş, Gül Sena Ehghaghi, Malikeh Lester, Brian Liu, Fengyuan Zhao, Wanru Ciccone, Marco Raffel, Colin Computation and Language Machine Learning Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we train fourteen models that use different tokenizers but are otherwise identical using the same architecture, dataset, training budget, and initialization. Additionally, we curate and release a new benchmark that specifically measures model performance subject to real-world perturbations that are likely to influence tokenization. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers. |
| title | TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2512.20757 |