REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908577351860224 |
|---|---|
| author | He, Qiyuan Li, Yicong Ye, Haotian Wang, Jinghao Liao, Xinyao Heng, Pheng-Ann Ermon, Stefano Zou, James Yao, Angela |
| author_facet | He, Qiyuan Li, Yicong Ye, Haotian Wang, Jinghao Liao, Xinyao Heng, Pheng-Ann Ermon, Stefano Zou, James Yao, Angela |
| contents | Visual autoregressive (AR) generation offers a promising path toward unifying vision and language models, yet its performance remains suboptimal against diffusion models. Prior work often attributes this gap to tokenizer limitations and rasterization ordering. In this work, we identify a core bottleneck from the perspective of generator-tokenizer inconsistency, i.e., the AR-generated tokens may not be well-decoded by the tokenizer. To address this, we propose reAR, a simple training strategy introducing a token-wise regularization objective: when predicting the next token, the causal transformer is also trained to recover the visual embedding of the current token and predict the embedding of the target token under a noisy context. It requires no changes to the tokenizer, generation order, inference pipeline, or external models. Despite its simplicity, reAR substantially improves performance. On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standard rasterization-based tokenizer. When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04450 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization He, Qiyuan Li, Yicong Ye, Haotian Wang, Jinghao Liao, Xinyao Heng, Pheng-Ann Ermon, Stefano Zou, James Yao, Angela Computer Vision and Pattern Recognition Visual autoregressive (AR) generation offers a promising path toward unifying vision and language models, yet its performance remains suboptimal against diffusion models. Prior work often attributes this gap to tokenizer limitations and rasterization ordering. In this work, we identify a core bottleneck from the perspective of generator-tokenizer inconsistency, i.e., the AR-generated tokens may not be well-decoded by the tokenizer. To address this, we propose reAR, a simple training strategy introducing a token-wise regularization objective: when predicting the next token, the causal transformer is also trained to recover the visual embedding of the current token and predict the embedding of the target token under a noisy context. It requires no changes to the tokenizer, generation order, inference pipeline, or external models. Despite its simplicity, reAR substantially improves performance. On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standard rasterization-based tokenizer. When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M). |
| title | REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.04450 |