Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.18869 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914959004270592 |
|---|---|
| author | Wang, Xinlong Zhang, Xiaosong Luo, Zhengxiong Sun, Quan Cui, Yufeng Wang, Jinsheng Zhang, Fan Wang, Yueze Li, Zhen Yu, Qiying Zhao, Yingli Ao, Yulong Min, Xuebin Li, Tao Wu, Boya Zhao, Bo Zhang, Bowen Wang, Liangdong Liu, Guang He, Zheqi Yang, Xi Liu, Jingjing Lin, Yonghua Huang, Tiejun Wang, Zhongyuan |
| author_facet | Wang, Xinlong Zhang, Xiaosong Luo, Zhengxiong Sun, Quan Cui, Yufeng Wang, Jinsheng Zhang, Fan Wang, Yueze Li, Zhen Yu, Qiying Zhao, Yingli Ao, Yulong Min, Xuebin Li, Tao Wu, Boya Zhao, Bo Zhang, Bowen Wang, Liangdong Liu, Guang He, Zheqi Yang, Xi Liu, Jingjing Lin, Yonghua Huang, Tiejun Wang, Zhongyuan |
| contents | While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18869 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Emu3: Next-Token Prediction is All You Need Wang, Xinlong Zhang, Xiaosong Luo, Zhengxiong Sun, Quan Cui, Yufeng Wang, Jinsheng Zhang, Fan Wang, Yueze Li, Zhen Yu, Qiying Zhao, Yingli Ao, Yulong Min, Xuebin Li, Tao Wu, Boya Zhao, Bo Zhang, Bowen Wang, Liangdong Liu, Guang He, Zheqi Yang, Xi Liu, Jingjing Lin, Yonghua Huang, Tiejun Wang, Zhongyuan Computer Vision and Pattern Recognition While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction. |
| title | Emu3: Next-Token Prediction is All You Need |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.18869 |