OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911195870527488 |
|---|---|
| author | Li, Han Peng, Xinyu Wang, Yaoming Peng, Zelin Chen, Xin Weng, Rongxiang Wang, Jingang Cai, Xunliang Dai, Wenrui Xiong, Hongkai |
| author_facet | Li, Han Peng, Xinyu Wang, Yaoming Peng, Zelin Chen, Xin Weng, Rongxiang Wang, Jingang Cai, Xunliang Dai, Wenrui Xiong, Hongkai |
| contents | We introduce OneCAT, a unified multimodal model that seamlessly integrates understanding, generation, and editing within a novel, pure decoder-only transformer architecture. Our framework uniquely eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference, leading to significant efficiency gains, especially for high-resolution inputs. This is achieved through a modality-specific Mixture-of-Experts (MoE) structure trained with a single autoregressive (AR) objective, which also natively supports dynamic resolutions. Furthermore, we pioneer a multi-scale visual autoregressive mechanism within the Large Language Model (LLM) that drastically reduces decoding steps compared to diffusion-based methods while maintaining state-of-the-art performance. Our findings demonstrate the powerful potential of pure autoregressive modeling as a sufficient and elegant foundation for unified multimodal intelligence. As a result, OneCAT sets a new performance standard, outperforming existing open-source unified multimodal models across benchmarks for multimodal generation, editing, and understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_03498 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation Li, Han Peng, Xinyu Wang, Yaoming Peng, Zelin Chen, Xin Weng, Rongxiang Wang, Jingang Cai, Xunliang Dai, Wenrui Xiong, Hongkai Computer Vision and Pattern Recognition We introduce OneCAT, a unified multimodal model that seamlessly integrates understanding, generation, and editing within a novel, pure decoder-only transformer architecture. Our framework uniquely eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference, leading to significant efficiency gains, especially for high-resolution inputs. This is achieved through a modality-specific Mixture-of-Experts (MoE) structure trained with a single autoregressive (AR) objective, which also natively supports dynamic resolutions. Furthermore, we pioneer a multi-scale visual autoregressive mechanism within the Large Language Model (LLM) that drastically reduces decoding steps compared to diffusion-based methods while maintaining state-of-the-art performance. Our findings demonstrate the powerful potential of pure autoregressive modeling as a sufficient and elegant foundation for unified multimodal intelligence. As a result, OneCAT sets a new performance standard, outperforming existing open-source unified multimodal models across benchmarks for multimodal generation, editing, and understanding. |
| title | OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.03498 |