MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917973242937344 |
|---|---|
| author | Li, Siyuan Zhang, Luyuan Wang, Zedong Tian, Juanxi Tan, Cheng Liu, Zicheng Yu, Chang Xie, Qingsong Lu, Haonan Wang, Haoqian Lei, Zhen |
| author_facet | Li, Siyuan Zhang, Luyuan Wang, Zedong Tian, Juanxi Tan, Cheng Liu, Zicheng Yu, Chang Xie, Qingsong Lu, Haonan Wang, Haoqian Lei, Zhen |
| contents | Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods struggle to address the trade-off in shared latent space for generation quality vs. representation learning and efficiency. To push the limits of this paradigm, we propose MergeVQ, which incorporates token merging techniques into VQ-based generative models to bridge the gap between image generation and visual representation learning in a unified architecture. During pre-training, MergeVQ decouples top-k semantics from latent space with the token merge module after self-attention blocks in the encoder for subsequent Look-up Free Quantization (LFQ) and global alignment and recovers their fine-grained details through cross-attention in the decoder for reconstruction. As for the second-stage generation, we introduce MergeAR, which performs KV Cache compression for efficient raster-order prediction. Extensive experiments on ImageNet verify that MergeVQ as an AR generative model achieves competitive performance in both visual representation learning and image generation tasks while maintaining favorable token efficiency and inference speed. The code and model will be available at https://apexgen-x.github.io/MergeVQ. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_00999 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization Li, Siyuan Zhang, Luyuan Wang, Zedong Tian, Juanxi Tan, Cheng Liu, Zicheng Yu, Chang Xie, Qingsong Lu, Haonan Wang, Haoqian Lei, Zhen Computer Vision and Pattern Recognition Artificial Intelligence Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation. However, most existing methods struggle to address the trade-off in shared latent space for generation quality vs. representation learning and efficiency. To push the limits of this paradigm, we propose MergeVQ, which incorporates token merging techniques into VQ-based generative models to bridge the gap between image generation and visual representation learning in a unified architecture. During pre-training, MergeVQ decouples top-k semantics from latent space with the token merge module after self-attention blocks in the encoder for subsequent Look-up Free Quantization (LFQ) and global alignment and recovers their fine-grained details through cross-attention in the decoder for reconstruction. As for the second-stage generation, we introduce MergeAR, which performs KV Cache compression for efficient raster-order prediction. Extensive experiments on ImageNet verify that MergeVQ as an AR generative model achieves competitive performance in both visual representation learning and image generation tasks while maintaining favorable token efficiency and inference speed. The code and model will be available at https://apexgen-x.github.io/MergeVQ. |
| title | MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.00999 |