Token Distillation: Attention-aware Input Embeddings For New Tokens
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912963898638336 |
|---|---|
| author | Dobler, Konstantin Elliott, Desmond de Melo, Gerard |
| author_facet | Dobler, Konstantin Elliott, Desmond de Melo, Gerard |
| contents | Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20133 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Token Distillation: Attention-aware Input Embeddings For New Tokens Dobler, Konstantin Elliott, Desmond de Melo, Gerard Computation and Language Machine Learning Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines. |
| title | Token Distillation: Attention-aware Input Embeddings For New Tokens |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.20133 |