MIO: A Foundation Model on Multimodal Tokens
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911212531351552 |
|---|---|
| author | Wang, Zekun Zhu, King Xu, Chunpu Zhou, Wangchunshu Liu, Jiaheng Zhang, Yibo Wang, Jiashuo Shi, Ning Li, Siyu Li, Yizhi Que, Haoran Zhang, Zhaoxiang Zhang, Yuanxing Zhang, Ge Xu, Ke Fu, Jie Huang, Wenhao |
| author_facet | Wang, Zekun Zhu, King Xu, Chunpu Zhou, Wangchunshu Liu, Jiaheng Zhang, Yibo Wang, Jiashuo Shi, Ning Li, Siyu Li, Yizhi Que, Haoran Zhang, Zhaoxiang Zhang, Yuanxing Zhang, Ge Xu, Ke Fu, Jie Huang, Wenhao |
| contents | In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17692 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MIO: A Foundation Model on Multimodal Tokens Wang, Zekun Zhu, King Xu, Chunpu Zhou, Wangchunshu Liu, Jiaheng Zhang, Yibo Wang, Jiashuo Shi, Ning Li, Siyu Li, Yizhi Que, Haoran Zhang, Zhaoxiang Zhang, Yuanxing Zhang, Ge Xu, Ke Fu, Jie Huang, Wenhao Computation and Language Artificial Intelligence Machine Learning In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. While the emergence of large language models (LLMs) and multimodal large language models (MM-LLMs) propels advancements in artificial general intelligence through their versatile capabilities, they still lack true any-to-any understanding and generation. Recently, the release of GPT-4o has showcased the remarkable potential of any-to-any LLMs for complex real-world tasks, enabling omnidirectional input and output across images, speech, and text. However, it is closed-source and does not support the generation of multimodal interleaved sequences. To address this gap, we present MIO, which is trained on a mixture of discrete tokens across four modalities using causal multimodal modeling. MIO undergoes a four-stage training process: (1) alignment pre-training, (2) interleaved pre-training, (3) speech-enhanced pre-training, and (4) comprehensive supervised fine-tuning on diverse textual, visual, and speech tasks. Our experimental results indicate that MIO exhibits competitive, and in some cases superior, performance compared to previous dual-modal baselines, any-to-any model baselines, and even modality-specific baselines. Moreover, MIO demonstrates advanced capabilities inherent to its any-to-any feature, such as interleaved video-text generation, chain-of-visual-thought reasoning, visual guideline generation, instructional image editing, etc. |
| title | MIO: A Foundation Model on Multimodal Tokens |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2409.17692 |