MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912088030445568 |
|---|---|
| author | Liu, Haozhe Liu, Shikun Zhou, Zijian Xu, Mengmeng Xie, Yanping Han, Xiao Pérez, Juan C. Liu, Ding Kahatapitiya, Kumara Jia, Menglin Wu, Jui-Chieh He, Sen Xiang, Tao Schmidhuber, Jürgen Pérez-Rúa, Juan-Manuel |
| author_facet | Liu, Haozhe Liu, Shikun Zhou, Zijian Xu, Mengmeng Xie, Yanping Han, Xiao Pérez, Juan C. Liu, Ding Kahatapitiya, Kumara Jia, Menglin Wu, Jui-Chieh He, Sen Xiang, Tao Schmidhuber, Jürgen Pérez-Rúa, Juan-Manuel |
| contents | We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_20280 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MarDini: Masked Autoregressive Diffusion for Video Generation at Scale Liu, Haozhe Liu, Shikun Zhou, Zijian Xu, Mengmeng Xie, Yanping Han, Xiao Pérez, Juan C. Liu, Ding Kahatapitiya, Kumara Jia, Menglin Wu, Jui-Chieh He, Sen Xiang, Tao Schmidhuber, Jürgen Pérez-Rúa, Juan-Manuel Computer Vision and Pattern Recognition Artificial Intelligence We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini's MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models. |
| title | MarDini: Masked Autoregressive Diffusion for Video Generation at Scale |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2410.20280 |