Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2605.25343 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914598964166656 |
|---|---|
| author | An, Siyu Lu, Junru Dong, Junnan Wang, Qiufeng Li, Yinghui Fei, Weizhi Yu, Zichao Yuan, Zheng Liu, Biao Wang, Haopeng Liang, Renzhao Yang, Yixuan Shen, Yunhang Ke, Bo Chen, Keyu Luo, Linhao Zou, Difan Huang, Xiao Yin, Di Qiao, Ruizhi Sun, Xing |
| author_facet | An, Siyu Lu, Junru Dong, Junnan Wang, Qiufeng Li, Yinghui Fei, Weizhi Yu, Zichao Yuan, Zheng Liu, Biao Wang, Haopeng Liang, Renzhao Yang, Yixuan Shen, Yunhang Ke, Bo Chen, Keyu Luo, Linhao Zou, Difan Huang, Xiao Yin, Di Qiao, Ruizhi Sun, Xing |
| contents | Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25343 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Toward Native Multimodal Modeling: A Roadmap An, Siyu Lu, Junru Dong, Junnan Wang, Qiufeng Li, Yinghui Fei, Weizhi Yu, Zichao Yuan, Zheng Liu, Biao Wang, Haopeng Liang, Renzhao Yang, Yixuan Shen, Yunhang Ke, Bo Chen, Keyu Luo, Linhao Zou, Difan Huang, Xiao Yin, Di Qiao, Ruizhi Sun, Xing Computer Vision and Pattern Recognition Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling. |
| title | Toward Native Multimodal Modeling: A Roadmap |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.25343 |