X-Fusion: Introducing New Modality to Frozen Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913812425211904 |
|---|---|
| author | Mo, Sicheng Nguyen, Thao Huang, Xun Iyer, Siddharth Srinivasan Li, Yijun Liu, Yuchen Tandon, Abhishek Shechtman, Eli Singh, Krishna Kumar Lee, Yong Jae Zhou, Bolei Li, Yuheng |
| author_facet | Mo, Sicheng Nguyen, Thao Huang, Xun Iyer, Siddharth Srinivasan Li, Yijun Liu, Yuchen Tandon, Abhishek Shechtman, Eli Singh, Krishna Kumar Lee, Yong Jae Zhou, Bolei Li, Yuheng |
| contents | We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation. Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks. We find that incorporating understanding-focused data improves generation quality, reducing image data noise enhances overall performance, and feature alignment accelerates convergence for smaller models but has minimal impact on larger ones. Our findings provide valuable insights into building efficient unified multimodal models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20996 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | X-Fusion: Introducing New Modality to Frozen Large Language Models Mo, Sicheng Nguyen, Thao Huang, Xun Iyer, Siddharth Srinivasan Li, Yijun Liu, Yuchen Tandon, Abhishek Shechtman, Eli Singh, Krishna Kumar Lee, Yong Jae Zhou, Bolei Li, Yuheng Computer Vision and Pattern Recognition We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation. Our experiments demonstrate that X-Fusion consistently outperforms alternative architectures on both image-to-text and text-to-image tasks. We find that incorporating understanding-focused data improves generation quality, reducing image data noise enhances overall performance, and feature alignment accelerates convergence for smaller models but has minimal impact on larger ones. Our findings provide valuable insights into building efficient unified multimodal models. |
| title | X-Fusion: Introducing New Modality to Frozen Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.20996 |