Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.05289 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910022596820992 |
|---|---|
| author | Wu, Pingyu Zhu, Kai Liu, Yu Tang, Longxiang Yang, Jian Peng, Yansong Zhai, Wei Cao, Yang Zha, Zheng-Jun |
| author_facet | Wu, Pingyu Zhu, Kai Liu, Yu Tang, Longxiang Yang, Jian Peng, Yansong Zhai, Wei Cao, Yang Zha, Zheng-Jun |
| contents | Autoregressive image generation aims to predict the next token based on previous ones. However, this process is challenged by the bidirectional dependencies inherent in conventional image tokenizations, which creates a fundamental misalignment with the unidirectional nature of autoregressive models. To resolve this, we introduce AliTok, a novel Aligned Tokenizer that alters the dependency structure of the token sequence. AliTok employs a bidirectional encoder constrained by a causal decoder, a design that compels the encoder to produce a token sequence with both semantic richness and forward-dependency. Furthermore, by incorporating prefix tokens and employing a two-stage tokenizer training process to enhance reconstruction performance, AliTok achieves high fidelity and predictability simultaneously. Building upon AliTok, a standard decoder-only autoregressive model with just 177M parameters achieves a gFID of 1.44 and an IS of 319.5 on ImageNet-256. Scaling to 662M, our model reaches a gFID of 1.28, surpassing the SOTA diffusion method with 10x faster sampling. On ImageNet-512, our 318M model also achieves a SOTA gFID of 1.39. Code and weights at https://github.com/ali-vilab/alitok. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05289 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model Wu, Pingyu Zhu, Kai Liu, Yu Tang, Longxiang Yang, Jian Peng, Yansong Zhai, Wei Cao, Yang Zha, Zheng-Jun Computer Vision and Pattern Recognition Autoregressive image generation aims to predict the next token based on previous ones. However, this process is challenged by the bidirectional dependencies inherent in conventional image tokenizations, which creates a fundamental misalignment with the unidirectional nature of autoregressive models. To resolve this, we introduce AliTok, a novel Aligned Tokenizer that alters the dependency structure of the token sequence. AliTok employs a bidirectional encoder constrained by a causal decoder, a design that compels the encoder to produce a token sequence with both semantic richness and forward-dependency. Furthermore, by incorporating prefix tokens and employing a two-stage tokenizer training process to enhance reconstruction performance, AliTok achieves high fidelity and predictability simultaneously. Building upon AliTok, a standard decoder-only autoregressive model with just 177M parameters achieves a gFID of 1.44 and an IS of 319.5 on ImageNet-256. Scaling to 662M, our model reaches a gFID of 1.28, surpassing the SOTA diffusion method with 10x faster sampling. On ImageNet-512, our 318M model also achieves a SOTA gFID of 1.39. Code and weights at https://github.com/ali-vilab/alitok. |
| title | Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.05289 |