Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913831289094144 |
|---|---|
| author | Liao, Chao Liu, Liyang Wang, Xun Luo, Zhengxiong Zhang, Xinyu Zhao, Wenliang Wu, Jie Li, Liang Tian, Zhi Huang, Weilin |
| author_facet | Liao, Chao Liu, Liyang Wang, Xun Luo, Zhengxiong Zhang, Xinyu Zhao, Wenliang Wu, Jie Li, Liang Tian, Zhi Huang, Weilin |
| contents | Recent progress in unified models for image understanding and generation has been impressive, yet most approaches remain limited to single-modal generation conditioned on multiple modalities. In this paper, we present Mogao, a unified framework that advances this paradigm by enabling interleaved multi-modal generation through a causal approach. Mogao integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance, which allow it to harness the strengths of both autoregressive models for text generation and diffusion models for high-quality image synthesis. These practical improvements also make Mogao particularly effective to process interleaved sequences of text and images arbitrarily. To further unlock the potential of unified models, we introduce an efficient training strategy on a large-scale, in-house dataset specifically curated for joint text and image generation. Extensive experiments show that Mogao not only achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs. Its emergent capabilities in zero-shot image editing and compositional generation highlight Mogao as a practical omni-modal foundation model, paving the way for future development and scaling the unified multi-modal systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05472 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation Liao, Chao Liu, Liyang Wang, Xun Luo, Zhengxiong Zhang, Xinyu Zhao, Wenliang Wu, Jie Li, Liang Tian, Zhi Huang, Weilin Computer Vision and Pattern Recognition Recent progress in unified models for image understanding and generation has been impressive, yet most approaches remain limited to single-modal generation conditioned on multiple modalities. In this paper, we present Mogao, a unified framework that advances this paradigm by enabling interleaved multi-modal generation through a causal approach. Mogao integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance, which allow it to harness the strengths of both autoregressive models for text generation and diffusion models for high-quality image synthesis. These practical improvements also make Mogao particularly effective to process interleaved sequences of text and images arbitrarily. To further unlock the potential of unified models, we introduce an efficient training strategy on a large-scale, in-house dataset specifically curated for joint text and image generation. Extensive experiments show that Mogao not only achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs. Its emergent capabilities in zero-shot image editing and compositional generation highlight Mogao as a practical omni-modal foundation model, paving the way for future development and scaling the unified multi-modal systems. |
| title | Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.05472 |