Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2406.16508 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908381686530048 |
|---|---|
| author | Takase, Sho Ri, Ryokan Kiyono, Shun Kato, Takuya |
| author_facet | Takase, Sho Ri, Ryokan Kiyono, Shun Kato, Takuya |
| contents | This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary sizes lead to better performance in LLMs. Moreover, we consider a continual training scenario where a pre-trained language model is trained on a different target language. We introduce a simple method to use a new vocabulary instead of the pre-defined one. We show that using the new vocabulary outperforms the model with the vocabulary used in pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16508 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Large Vocabulary Size Improves Large Language Models Takase, Sho Ri, Ryokan Kiyono, Shun Kato, Takuya Computation and Language This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary sizes lead to better performance in LLMs. Moreover, we consider a continual training scenario where a pre-trained language model is trained on a different target language. We introduce a simple method to use a new vocabulary instead of the pre-defined one. We show that using the new vocabulary outperforms the model with the vocabulary used in pre-training. |
| title | Large Vocabulary Size Improves Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.16508 |