Grid: Omni Visual Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908427915100160 |
|---|---|
| author | Wan, Cong Luo, Xiangyang Luo, Hao Cai, Zijian Song, Yiren Zhao, Yunlong Bai, Yifan Wang, Fan He, Yuhang Gong, Yihong |
| author_facet | Wan, Cong Luo, Xiangyang Luo, Hao Cai, Zijian Song, Yiren Zhao, Yunlong Bai, Yifan Wang, Fan He, Yuhang Gong, Yihong |
| contents | Visual generation has witnessed remarkable progress in single-image tasks, yet extending these capabilities to temporal sequences remains challenging. Current approaches either build specialized video models from scratch with enormous computational costs or add separate motion modules to image generators, both requiring learning temporal dynamics anew. We observe that modern image generation models possess underutilized potential in handling structured layouts with implicit temporal understanding. Building on this insight, we introduce GRID, which reformulates temporal sequences as grid layouts, enabling holistic processing of visual sequences while leveraging existing model capabilities. Through a parallel flow-matching training strategy with coarse-to-fine scheduling, our approach achieves up to 67 faster inference speeds while using <1/1000 of the computational resources compared to specialized models. Extensive experiments demonstrate that GRID not only excels in temporal tasks from Text-to-Video to 3D Editing but also preserves strong performance in image generation, establishing itself as an efficient and versatile omni-solution for visual generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10718 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Grid: Omni Visual Generation Wan, Cong Luo, Xiangyang Luo, Hao Cai, Zijian Song, Yiren Zhao, Yunlong Bai, Yifan Wang, Fan He, Yuhang Gong, Yihong Computer Vision and Pattern Recognition Visual generation has witnessed remarkable progress in single-image tasks, yet extending these capabilities to temporal sequences remains challenging. Current approaches either build specialized video models from scratch with enormous computational costs or add separate motion modules to image generators, both requiring learning temporal dynamics anew. We observe that modern image generation models possess underutilized potential in handling structured layouts with implicit temporal understanding. Building on this insight, we introduce GRID, which reformulates temporal sequences as grid layouts, enabling holistic processing of visual sequences while leveraging existing model capabilities. Through a parallel flow-matching training strategy with coarse-to-fine scheduling, our approach achieves up to 67 faster inference speeds while using <1/1000 of the computational resources compared to specialized models. Extensive experiments demonstrate that GRID not only excels in temporal tasks from Text-to-Video to 3D Editing but also preserves strong performance in image generation, establishing itself as an efficient and versatile omni-solution for visual generation. |
| title | Grid: Omni Visual Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.10718 |