MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917547376377856 |
|---|---|
| author | Zhang, Luyuan Li, Siyuan Wang, Zedong Xie, Qingsong Tan, Cheng Wang, Anna Zhang, Yanhao Chen, Chen Lu, Haonan Wang, Haoqian |
| author_facet | Zhang, Luyuan Li, Siyuan Wang, Zedong Xie, Qingsong Tan, Cheng Wang, Anna Zhang, Yanhao Chen, Chen Lu, Haonan Wang, Haoqian |
| contents | Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_30904 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging Zhang, Luyuan Li, Siyuan Wang, Zedong Xie, Qingsong Tan, Cheng Wang, Anna Zhang, Yanhao Chen, Chen Lu, Haonan Wang, Haoqian Computer Vision and Pattern Recognition Most visual tokenizers for image generation are bifurcated into two families with complementary limitations: continuous VAEs offer high-fidelity reconstruction but suffer from dense, entangled latents that are poorly suited for semantic control, whereas discrete VQ-based models enable autoregressive generation yet struggle with gradient sparsity, unstable training, and codebook collapse. In this work, we introduce MergeTok, a unified tokenizer that jointly optimizes continuous (VAE) and discrete (VQ) tokenizers within a encoder-decoder architecture, leveraging token merging techniques as a semantic bridge. By clustering similar tokens during encoding, MergeTok establishes a structural prior that provides dual supervision signals: (i) it imposes merged-token semantic alignment in the VAE branch, regularizing its latent space toward disentangled, semantic-aware representations; (ii) it derives group-wise constraints, promoting intra-group diversity and inter-group exclusivity that stabilize VQ training. MergeTok shows competitive reconstruction and generation performance on ImageNet-256, with substantially lower rFID than strong VAE and VQ models under matched token budgets, while producing semantically-organized token representations compatible with both autoregressive and diffusion generators. This shows that a single architecture can endow visual tokenizers with robust semantic organization and generator-friendly discreteness. |
| title | MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.30904 |