Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916798963646464 |
|---|---|
| author | Cho, Gyeongje So, Yeonkyoun Park, Chanwoo Lee, Sangmin Jung, Sungmok Lee, Jaejin |
| author_facet | Cho, Gyeongje So, Yeonkyoun Park, Chanwoo Lee, Sangmin Jung, Sungmok Lee, Jaejin |
| contents | This paper introduces Thunder-Tok, a new Korean tokenizer designed to reduce token fertility without compromising model performance. Our approach uses a rule-based pre-tokenization method that aligns with the linguistic structure of the Korean language. We also create a seed vocabulary containing tokens that resemble linguistic units and employ a branching entropy-based selection algorithm. These techniques increase the average token length, thus lowering fertility while preserving linguistic information. Experimental results indicate that Thunder-Tok reduces fertility by approximately 10% (i.e., reduces the number of tokens by 10%, improving the inference speed by 10%) compared to BPE without compromising performance across various downstream tasks. These findings demonstrate that our linguistically informed approach is effective and practical for designing efficient tokenizers for language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15138 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models Cho, Gyeongje So, Yeonkyoun Park, Chanwoo Lee, Sangmin Jung, Sungmok Lee, Jaejin Computation and Language Artificial Intelligence This paper introduces Thunder-Tok, a new Korean tokenizer designed to reduce token fertility without compromising model performance. Our approach uses a rule-based pre-tokenization method that aligns with the linguistic structure of the Korean language. We also create a seed vocabulary containing tokens that resemble linguistic units and employ a branching entropy-based selection algorithm. These techniques increase the average token length, thus lowering fertility while preserving linguistic information. Experimental results indicate that Thunder-Tok reduces fertility by approximately 10% (i.e., reduces the number of tokens by 10%, improving the inference speed by 10%) compared to BPE without compromising performance across various downstream tasks. These findings demonstrate that our linguistically informed approach is effective and practical for designing efficient tokenizers for language models. |
| title | Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.15138 |