UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911308682625024 |
|---|---|
| author | Huang, Jiehui Zhang, Yuechen He, Xu Gao, Yuan Cen, Zhi Xia, Bin Zhou, Yan Tao, Xin Wan, Pengfei Jia, Jiaya |
| author_facet | Huang, Jiehui Zhang, Yuechen He, Xu Gao, Yuan Cen, Zhi Xia, Bin Zhou, Yan Tao, Xin Wan, Pengfei Jia, Jiaya |
| contents | Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: https://github.com/dvlab-research/UnityVideo |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_07831 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation Huang, Jiehui Zhang, Yuechen He, Xu Gao, Yuan Cen, Zhi Xia, Bin Zhou, Yan Tao, Xin Wan, Pengfei Jia, Jiaya Computer Vision and Pattern Recognition Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: https://github.com/dvlab-research/UnityVideo |
| title | UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.07831 |