HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909635088220160 |
|---|---|
| author | Xiao, Yicheng Song, Lin Yang, Rui Cheng, Cheng Xu, Zunnan Zhang, Zhaoyang Ge, Yixiao Li, Xiu Shan, Ying |
| author_facet | Xiao, Yicheng Song, Lin Yang, Rui Cheng, Cheng Xu, Zunnan Zhang, Zhaoyang Ge, Yixiao Li, Xiu Shan, Ying |
| contents | With the advancement of language models, unified multimodal understanding and generation have made significant strides, with model architectures evolving from separated components to unified single-model frameworks. This paper explores an efficient training paradigm to build a single transformer for unified multimodal understanding and generation. Specifically, we propose a multimodal warmup strategy utilizing prior knowledge to extend capabilities. To address cross-modal compatibility challenges, we introduce feature pre-scaling and multimodal AdaLN techniques. Integrating the proposed technologies, we present the HaploOmni, a new single multimodal transformer. With limited training costs, HaploOmni achieves competitive performance across multiple image and video understanding and generation benchmarks over advanced unified models. All codes will be made public at https://github.com/Tencent/HaploVLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02975 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation Xiao, Yicheng Song, Lin Yang, Rui Cheng, Cheng Xu, Zunnan Zhang, Zhaoyang Ge, Yixiao Li, Xiu Shan, Ying Computer Vision and Pattern Recognition Artificial Intelligence With the advancement of language models, unified multimodal understanding and generation have made significant strides, with model architectures evolving from separated components to unified single-model frameworks. This paper explores an efficient training paradigm to build a single transformer for unified multimodal understanding and generation. Specifically, we propose a multimodal warmup strategy utilizing prior knowledge to extend capabilities. To address cross-modal compatibility challenges, we introduce feature pre-scaling and multimodal AdaLN techniques. Integrating the proposed technologies, we present the HaploOmni, a new single multimodal transformer. With limited training costs, HaploOmni achieves competitive performance across multiple image and video understanding and generation benchmarks over advanced unified models. All codes will be made public at https://github.com/Tencent/HaploVLM. |
| title | HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2506.02975 |