Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912999478919168 |
|---|---|
| author | Chen, Daiwei Fu, Zhoutong Jiang, Chengming Zhang, Haichao Zhou, Ran Wang, Tan Yao, Chunnan Li, Guoyao Cai, Rui Cao, Yihan Jiang, Ruijie Borisyuk, Fedor Shen, Jianqiang Wu, Jingwei Vinayak, Ramya Korlakai |
| author_facet | Chen, Daiwei Fu, Zhoutong Jiang, Chengming Zhang, Haichao Zhou, Ran Wang, Tan Yao, Chunnan Li, Guoyao Cai, Rui Cao, Yihan Jiang, Ruijie Borisyuk, Fedor Shen, Jianqiang Wu, Jingwei Vinayak, Ramya Korlakai |
| contents | Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that \emph{token initialization} is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the \emph{Grounded Token Initialization Hypothesis}: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02324 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation Chen, Daiwei Fu, Zhoutong Jiang, Chengming Zhang, Haichao Zhou, Ran Wang, Tan Yao, Chunnan Li, Guoyao Cai, Rui Cao, Yihan Jiang, Ruijie Borisyuk, Fedor Shen, Jianqiang Wu, Jingwei Vinayak, Ramya Korlakai Computation and Language Artificial Intelligence Machine Learning Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that \emph{token initialization} is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the \emph{Grounded Token Initialization Hypothesis}: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension. |
| title | Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.02324 |