Unified Autoregressive Visual Generation and Understanding with Continuous Tokens
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917959212990464 |
|---|---|
| author | Fan, Lijie Tang, Luming Qin, Siyang Li, Tianhong Yang, Xuan Qiao, Siyuan Steiner, Andreas Sun, Chen Li, Yuanzhen Zhu, Tao Rubinstein, Michael Raptis, Michalis Sun, Deqing Soricut, Radu |
| author_facet | Fan, Lijie Tang, Luming Qin, Siyang Li, Tianhong Yang, Xuan Qiao, Siyuan Steiner, Andreas Sun, Chen Li, Yuanzhen Zhu, Tao Rubinstein, Michael Raptis, Michalis Sun, Deqing Soricut, Radu |
| contents | We present UniFluid, a unified autoregressive framework for joint visual generation and understanding leveraging continuous visual tokens. Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image. We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other. By selecting an appropriate loss balance weight, the unified model achieves results comparable to or exceeding those of single-task baselines on both tasks. Furthermore, we demonstrate that employing stronger pre-trained LLMs and random-order generation during training is important to achieve high-fidelity image generation within this unified framework. Built upon the Gemma model series, UniFluid exhibits competitive performance across both image generation and understanding, demonstrating strong transferability to various downstream tasks, including image editing for generation, as well as visual captioning and question answering for understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_13436 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unified Autoregressive Visual Generation and Understanding with Continuous Tokens Fan, Lijie Tang, Luming Qin, Siyang Li, Tianhong Yang, Xuan Qiao, Siyuan Steiner, Andreas Sun, Chen Li, Yuanzhen Zhu, Tao Rubinstein, Michael Raptis, Michalis Sun, Deqing Soricut, Radu Computer Vision and Pattern Recognition Machine Learning We present UniFluid, a unified autoregressive framework for joint visual generation and understanding leveraging continuous visual tokens. Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image. We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other. By selecting an appropriate loss balance weight, the unified model achieves results comparable to or exceeding those of single-task baselines on both tasks. Furthermore, we demonstrate that employing stronger pre-trained LLMs and random-order generation during training is important to achieve high-fidelity image generation within this unified framework. Built upon the Gemma model series, UniFluid exhibits competitive performance across both image generation and understanding, demonstrating strong transferability to various downstream tasks, including image editing for generation, as well as visual captioning and question answering for understanding. |
| title | Unified Autoregressive Visual Generation and Understanding with Continuous Tokens |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2503.13436 |