Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908848277684224 |
|---|---|
| author | Liu, Jialun Li, Tian Cao, Xiao Ma, Yukuo Shang, Gonghu Huang, Haibin Zhang, Chi Chang, Xiangzhen Huang, Zhiyong Hu, Jiakui Li, Zuoxin Liang, Yuanzhi Liu, Cong Liu, Junqi Tan, Robby T. Tang, Haitong Weng, Qizhen Xu, Yifan Yang, Liying Yang, Xiaoyan Yu, Peng Zhang, Shiwen Li, Xuelong |
| author_facet | Liu, Jialun Li, Tian Cao, Xiao Ma, Yukuo Shang, Gonghu Huang, Haibin Zhang, Chi Chang, Xiangzhen Huang, Zhiyong Hu, Jiakui Li, Zuoxin Liang, Yuanzhi Liu, Cong Liu, Junqi Tan, Robby T. Tang, Haitong Weng, Qizhen Xu, Yifan Yang, Liying Yang, Xiaoyan Yu, Peng Zhang, Shiwen Li, Xuelong |
| contents | Recent advances in diffusion-based video generation have substantially improved visual fidelity and temporal coherence. However, most existing approaches remain task-specific and rely primarily on textual instructions, limiting their ability to handle multimodal inputs, contextual references, and diverse video generation and editing scenarios within a unified framework. Moreover, many video editing methods depend on carefully engineered pipelines tailored to individual operations, which hinders scalability and composability. In this paper, we propose Tele-Omni, a unified multimodal framework for video generation and editing that follows multimodal instructions, including text, images, and reference videos, within a single model. Tele-Omni leverages pretrained multimodal large language models to parse heterogeneous instructions and infer structured generation or editing intents, while diffusion-based generators perform high-quality video synthesis conditioned on these structured signals. To enable joint training across heterogeneous video tasks, we introduce a task-aware data processing pipeline that unifies multimodal inputs into a structured instruction format while preserving task-specific constraints. Tele-Omni supports a wide range of video-centric tasks, including text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing. By decoupling instruction parsing from video synthesis and combining it with task-aware data design, Tele-Omni achieves flexible multimodal control while maintaining strong temporal coherence and visual consistency. Experimental results demonstrate that Tele-Omni achieves competitive performance across multiple tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09609 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing Liu, Jialun Li, Tian Cao, Xiao Ma, Yukuo Shang, Gonghu Huang, Haibin Zhang, Chi Chang, Xiangzhen Huang, Zhiyong Hu, Jiakui Li, Zuoxin Liang, Yuanzhi Liu, Cong Liu, Junqi Tan, Robby T. Tang, Haitong Weng, Qizhen Xu, Yifan Yang, Liying Yang, Xiaoyan Yu, Peng Zhang, Shiwen Li, Xuelong Computer Vision and Pattern Recognition Recent advances in diffusion-based video generation have substantially improved visual fidelity and temporal coherence. However, most existing approaches remain task-specific and rely primarily on textual instructions, limiting their ability to handle multimodal inputs, contextual references, and diverse video generation and editing scenarios within a unified framework. Moreover, many video editing methods depend on carefully engineered pipelines tailored to individual operations, which hinders scalability and composability. In this paper, we propose Tele-Omni, a unified multimodal framework for video generation and editing that follows multimodal instructions, including text, images, and reference videos, within a single model. Tele-Omni leverages pretrained multimodal large language models to parse heterogeneous instructions and infer structured generation or editing intents, while diffusion-based generators perform high-quality video synthesis conditioned on these structured signals. To enable joint training across heterogeneous video tasks, we introduce a task-aware data processing pipeline that unifies multimodal inputs into a structured instruction format while preserving task-specific constraints. Tele-Omni supports a wide range of video-centric tasks, including text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing. By decoupling instruction parsing from video synthesis and combining it with task-aware data design, Tele-Omni achieves flexible multimodal control while maintaining strong temporal coherence and visual consistency. Experimental results demonstrate that Tele-Omni achieves competitive performance across multiple tasks. |
| title | Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.09609 |