Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914404542447616 |
|---|---|
| author | Zhao, Wenhao Zou, Qiran Shah, Rushi Wu, Yudi Lin, Zhouhan Liu, Dianbo |
| author_facet | Zhao, Wenhao Zou, Qiran Shah, Rushi Wu, Yudi Lin, Zhouhan Liu, Dianbo |
| contents | Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we systematically investigate the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that random initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_17052 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization Zhao, Wenhao Zou, Qiran Shah, Rushi Wu, Yudi Lin, Zhouhan Liu, Dianbo Machine Learning Artificial Intelligence Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we systematically investigate the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that random initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization. |
| title | Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.17052 |