Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.22946 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908666372816896 |
|---|---|
| author | Wang, Zeyu Chen, Zilong Gou, Chenhui Li, Feng Deng, Chaorui Zhu, Deyao Li, Kunchang Yu, Weihao Tu, Haoqin Fan, Haoqi Xie, Cihang |
| author_facet | Wang, Zeyu Chen, Zilong Gou, Chenhui Li, Feng Deng, Chaorui Zhu, Deyao Li, Kunchang Yu, Weihao Tu, Haoqin Fan, Haoqi Xie, Cihang |
| contents | Unified multimodal models have recently shown remarkable gains in both capability and versatility, yet most leading systems are still trained from scratch and require substantial computational resources. In this paper, we show that competitive performance can be obtained far more efficiently by strategically fusing publicly available models specialized for either generation or understanding. Our key design is to retain the original blocks while additionally interleaving multimodal self-attention blocks throughout the networks. This double fusion mechanism (1) effectively enables rich multi-modal fusion while largely preserving the original strengths of the base models, and (2) catalyzes synergistic fusion of high-level semantic representations from the understanding encoder with low-level spatial signals from the generation encoder. By training with only ~ 35B tokens, this approach achieves strong results across multiple benchmarks: 0.91 on GenEval for compositional text-to-image generation, 82.16 on DPG-Bench for complex text-to-image generation, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench for image editing. By fully releasing the entire suite of code, model weights, and datasets, we hope to support future research on unified multimodal modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22946 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation Wang, Zeyu Chen, Zilong Gou, Chenhui Li, Feng Deng, Chaorui Zhu, Deyao Li, Kunchang Yu, Weihao Tu, Haoqin Fan, Haoqi Xie, Cihang Computer Vision and Pattern Recognition Unified multimodal models have recently shown remarkable gains in both capability and versatility, yet most leading systems are still trained from scratch and require substantial computational resources. In this paper, we show that competitive performance can be obtained far more efficiently by strategically fusing publicly available models specialized for either generation or understanding. Our key design is to retain the original blocks while additionally interleaving multimodal self-attention blocks throughout the networks. This double fusion mechanism (1) effectively enables rich multi-modal fusion while largely preserving the original strengths of the base models, and (2) catalyzes synergistic fusion of high-level semantic representations from the understanding encoder with low-level spatial signals from the generation encoder. By training with only ~ 35B tokens, this approach achieves strong results across multiple benchmarks: 0.91 on GenEval for compositional text-to-image generation, 82.16 on DPG-Bench for complex text-to-image generation, 6.06 on GEditBench, and 3.77 on ImgEdit-Bench for image editing. By fully releasing the entire suite of code, model weights, and datasets, we hope to support future research on unified multimodal modeling. |
| title | LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.22946 |