Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.21933 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909976268636160 |
|---|---|
| author | Pawar, Sachin Apte, Manoj Jadhav, Kshitij Palshikar, Girish Keshav Ramrakhiyani, Nitin |
| author_facet | Pawar, Sachin Apte, Manoj Jadhav, Kshitij Palshikar, Girish Keshav Ramrakhiyani, Nitin |
| contents | Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model's fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of "natural" words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral's tokenizer splits "martial" into "mart" and "ial"). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_21933 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs Pawar, Sachin Apte, Manoj Jadhav, Kshitij Palshikar, Girish Keshav Ramrakhiyani, Nitin Computation and Language Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model's fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of "natural" words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral's tokenizer splits "martial" into "mart" and "ial"). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs. |
| title | Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.21933 |