M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912312610258944 |
|---|---|
| author | Guo, Qingpei Song, Kaiyou Feng, Zipeng Ma, Ziping Zhang, Qinglong Gao, Sirui Yu, Xuzheng Sun, Yunxiao Chang, Tai-Wei Chen, Jingdong Yang, Ming Zhou, Jun |
| author_facet | Guo, Qingpei Song, Kaiyou Feng, Zipeng Ma, Ziping Zhang, Qinglong Gao, Sirui Yu, Xuzheng Sun, Yunxiao Chang, Tai-Wei Chen, Jingdong Yang, Ming Zhou, Jun |
| contents | We present M2-omni, a cutting-edge, open-source omni-MLLM that achieves competitive performance to GPT-4o. M2-omni employs a unified multimodal sequence modeling framework, which empowers Large Language Models(LLMs) to acquire comprehensive cross-modal understanding and generation capabilities. Specifically, M2-omni can process arbitrary combinations of audio, video, image, and text modalities as input, generating multimodal sequences interleaving with audio, image, or text outputs, thereby enabling an advanced and interactive real-time experience. The training of such an omni-MLLM is challenged by significant disparities in data quantity and convergence rates across modalities. To address these challenges, we propose a step balance strategy during pre-training to handle the quantity disparities in modality-specific data. Additionally, a dynamically adaptive balance strategy is introduced during the instruction tuning stage to synchronize the modality-wise training progress, ensuring optimal convergence. Notably, we prioritize preserving strong performance on pure text tasks to maintain the robustness of M2-omni's language understanding capability throughout the training process. To our best knowledge, M2-omni is currently a very competitive open-source model to GPT-4o, characterized by its comprehensive modality and task support, as well as its exceptional performance. We expect M2-omni will advance the development of omni-MLLMs, thus facilitating future research in this domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18778 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance Guo, Qingpei Song, Kaiyou Feng, Zipeng Ma, Ziping Zhang, Qinglong Gao, Sirui Yu, Xuzheng Sun, Yunxiao Chang, Tai-Wei Chen, Jingdong Yang, Ming Zhou, Jun Machine Learning Artificial Intelligence Computation and Language We present M2-omni, a cutting-edge, open-source omni-MLLM that achieves competitive performance to GPT-4o. M2-omni employs a unified multimodal sequence modeling framework, which empowers Large Language Models(LLMs) to acquire comprehensive cross-modal understanding and generation capabilities. Specifically, M2-omni can process arbitrary combinations of audio, video, image, and text modalities as input, generating multimodal sequences interleaving with audio, image, or text outputs, thereby enabling an advanced and interactive real-time experience. The training of such an omni-MLLM is challenged by significant disparities in data quantity and convergence rates across modalities. To address these challenges, we propose a step balance strategy during pre-training to handle the quantity disparities in modality-specific data. Additionally, a dynamically adaptive balance strategy is introduced during the instruction tuning stage to synchronize the modality-wise training progress, ensuring optimal convergence. Notably, we prioritize preserving strong performance on pure text tasks to maintain the robustness of M2-omni's language understanding capability throughout the training process. To our best knowledge, M2-omni is currently a very competitive open-source model to GPT-4o, characterized by its comprehensive modality and task support, as well as its exceptional performance. We expect M2-omni will advance the development of omni-MLLMs, thus facilitating future research in this domain. |
| title | M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.18778 |