Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.12560 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913701909495808 |
|---|---|
| author | Seo, Jean Kim, Jaeyoon Byun, SungJoo Shin, Hyopil |
| author_facet | Seo, Jean Kim, Jaeyoon Byun, SungJoo Shin, Hyopil |
| contents | The necessity of language-specific tokenizers intuitively appears crucial for effective natural language processing, yet empirical analyses on their significance and underlying reasons are lacking. This study explores how language-specific tokenizers influence the behavior of Large Language Models predominantly trained with English text data, through the case study of Korean. The research unfolds in two main stages: (1) the development of a Korean-specific extended tokenizer and (2) experiments to compare models with the basic tokenizer and the extended tokenizer through various Next Token Prediction tasks. Our in-depth analysis reveals that the extended tokenizer decreases confidence in incorrect predictions during generation and reduces cross-entropy in complex tasks, indicating a tendency to produce less nonsensical outputs. Consequently, the extended tokenizer provides stability during generation, potentially leading to higher performance in downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12560 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How does a Language-Specific Tokenizer affect LLMs? Seo, Jean Kim, Jaeyoon Byun, SungJoo Shin, Hyopil Computation and Language The necessity of language-specific tokenizers intuitively appears crucial for effective natural language processing, yet empirical analyses on their significance and underlying reasons are lacking. This study explores how language-specific tokenizers influence the behavior of Large Language Models predominantly trained with English text data, through the case study of Korean. The research unfolds in two main stages: (1) the development of a Korean-specific extended tokenizer and (2) experiments to compare models with the basic tokenizer and the extended tokenizer through various Next Token Prediction tasks. Our in-depth analysis reveals that the extended tokenizer decreases confidence in incorrect predictions during generation and reduces cross-entropy in complex tasks, indicating a tendency to produce less nonsensical outputs. Consequently, the extended tokenizer provides stability during generation, potentially leading to higher performance in downstream tasks. |
| title | How does a Language-Specific Tokenizer affect LLMs? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12560 |