ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909337190924288 |
|---|---|
| author | Gu, Shuhao Zhao, Mengdi Zhang, Bowen Wang, Liangdong Li, Jijie Liu, Guang |
| author_facet | Gu, Shuhao Zhao, Mengdi Zhang, Bowen Wang, Liangdong Li, Jijie Liu, Guang |
| contents | Tokenizer is an essential component for large language models (LLMs), and a tokenizer with a high compression rate can improve the model's representation and processing efficiency. However, the tokenizer cannot ensure high compression rate in all scenarios, and an increase in the average input and output lengths will increases the training and inference costs of the model. Therefore, it is crucial to find ways to improve the model's efficiency with minimal cost while maintaining the model's performance. In this work, we propose a method to improve model representation and processing efficiency by replacing the tokenizers of LLMs. We propose replacing and reinitializing the parameters of the model's input and output layers with the parameters of the original model, and training these parameters while keeping other parameters fixed. We conducted experiments on different LLMs, and the results show that our method can maintain the performance of the model after replacing the tokenizer, while significantly improving the decoding speed for long texts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04335 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model Gu, Shuhao Zhao, Mengdi Zhang, Bowen Wang, Liangdong Li, Jijie Liu, Guang Computation and Language Tokenizer is an essential component for large language models (LLMs), and a tokenizer with a high compression rate can improve the model's representation and processing efficiency. However, the tokenizer cannot ensure high compression rate in all scenarios, and an increase in the average input and output lengths will increases the training and inference costs of the model. Therefore, it is crucial to find ways to improve the model's efficiency with minimal cost while maintaining the model's performance. In this work, we propose a method to improve model representation and processing efficiency by replacing the tokenizers of LLMs. We propose replacing and reinitializing the parameters of the model's input and output layers with the parameters of the original model, and training these parameters while keeping other parameters fixed. We conducted experiments on different LLMs, and the results show that our method can maintain the performance of the model after replacing the tokenizer, while significantly improving the decoding speed for long texts. |
| title | ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.04335 |