Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2501.16975 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916753639997440 |
|---|---|
| author | Huang, Hongzhi Zhu, Defa Wu, Banggu Zeng, Yutao Wang, Ya Min, Qiyang Zhou, Xun |
| author_facet | Huang, Hongzhi Zhu, Defa Wu, Banggu Zeng, Yutao Wang, Ya Min, Qiyang Zhou, Xun |
| contents | Tokenization is a fundamental component of large language models (LLMs), yet its influence on model scaling and performance is not fully explored. In this paper, we introduce Over-Tokenized Transformers, a novel framework that decouples input and output vocabularies to improve language modeling performance. Specifically, our approach scales up input vocabularies to leverage multi-gram tokens. Through extensive experiments, we uncover a log-linear relationship between input vocabulary size and training loss, demonstrating that larger input vocabularies consistently enhance model performance, regardless of model size. Using a large input vocabulary, we achieve performance comparable to double-sized baselines with no additional cost. Our findings highlight the importance of tokenization in scaling laws and provide practical insight for tokenizer design, paving the way for more efficient and powerful LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16975 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling Huang, Hongzhi Zhu, Defa Wu, Banggu Zeng, Yutao Wang, Ya Min, Qiyang Zhou, Xun Computation and Language Machine Learning Tokenization is a fundamental component of large language models (LLMs), yet its influence on model scaling and performance is not fully explored. In this paper, we introduce Over-Tokenized Transformers, a novel framework that decouples input and output vocabularies to improve language modeling performance. Specifically, our approach scales up input vocabularies to leverage multi-gram tokens. Through extensive experiments, we uncover a log-linear relationship between input vocabulary size and training loss, demonstrating that larger input vocabularies consistently enhance model performance, regardless of model size. Using a large input vocabulary, we achieve performance comparable to double-sized baselines with no additional cost. Our findings highlight the importance of tokenization in scaling laws and provide practical insight for tokenizer design, paving the way for more efficient and powerful LLMs. |
| title | Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2501.16975 |