Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.11694 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929740192940032 |
|---|---|
| author | Jiang, Shixin Liang, Jiafeng Wang, Jiyuan Dong, Xuan Chang, Heng Yu, Weijiang Du, Jinhua Liu, Ming Qin, Bing |
| author_facet | Jiang, Shixin Liang, Jiafeng Wang, Jiyuan Dong, Xuan Chang, Heng Yu, Weijiang Du, Jinhua Liu, Ming Qin, Bing |
| contents | To tackle complex tasks in real-world scenarios, more researchers are focusing on Omni-MLLMs, which aim to achieve omni-modal understanding and generation. Beyond the constraints of any specific non-linguistic modality, Omni-MLLMs map various non-linguistic modalities into the embedding space of LLMs and enable the interaction and understanding of arbitrary combinations of modalities within a single model. In this paper, we systematically investigate relevant research and provide a comprehensive survey of Omni-MLLMs. Specifically, we first explain the four core components of Omni-MLLMs for unified multi-modal modeling with a meticulous taxonomy that offers novel perspectives. Then, we introduce the effective integration achieved through two-stage training and discuss the corresponding datasets as well as evaluation. Furthermore, we summarize the main challenges of current Omni-MLLMs and outline future directions. We hope this paper serves as an introduction for beginners and promotes the advancement of related research. Resources have been made publicly available at https://github.com/threegold116/Awesome-Omni-MLLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11694 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities Jiang, Shixin Liang, Jiafeng Wang, Jiyuan Dong, Xuan Chang, Heng Yu, Weijiang Du, Jinhua Liu, Ming Qin, Bing Artificial Intelligence Computation and Language Machine Learning To tackle complex tasks in real-world scenarios, more researchers are focusing on Omni-MLLMs, which aim to achieve omni-modal understanding and generation. Beyond the constraints of any specific non-linguistic modality, Omni-MLLMs map various non-linguistic modalities into the embedding space of LLMs and enable the interaction and understanding of arbitrary combinations of modalities within a single model. In this paper, we systematically investigate relevant research and provide a comprehensive survey of Omni-MLLMs. Specifically, we first explain the four core components of Omni-MLLMs for unified multi-modal modeling with a meticulous taxonomy that offers novel perspectives. Then, we introduce the effective integration achieved through two-stage training and discuss the corresponding datasets as well as evaluation. Furthermore, we summarize the main challenges of current Omni-MLLMs and outline future directions. We hope this paper serves as an introduction for beginners and promotes the advancement of related research. Resources have been made publicly available at https://github.com/threegold116/Awesome-Omni-MLLMs. |
| title | From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2412.11694 |