Autoregressive Video Generation beyond Next Frames Prediction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915520207388672 |
|---|---|
| author | Ren, Sucheng Chen, Chen Wang, Zhenbang Song, Liangchen Zhu, Xiangxin Yuille, Alan Yang, Yinfei Lu, Jiasen |
| author_facet | Ren, Sucheng Chen, Chen Wang, Zhenbang Song, Liangchen Zhu, Xiangxin Yuille, Alan Yang, Yinfei Lu, Jiasen |
| contents | Autoregressive models for video generation typically operate frame-by-frame, extending next-token prediction from language to video's temporal dimension. We question that unlike word as token is universally agreed in language if frame is a appropriate prediction unit? To address this, we present VideoAR, a unified framework that supports a spectrum of prediction units including full frames, key-detail frames, multiscale refinements, and spatiotemporal cubes. Among these designs, we find model video generation using \textit{spatiotemporal} cubes as prediction units, which allows autoregressive models to operate across both spatial and temporal dimensions simultaneously. This approach eliminates the assumption that frames are the natural atomic units for video autoregression. We evaluate VideoAR across diverse prediction strategies, finding that cube-based prediction consistently delivers superior quality, speed, and temporal coherence. By removing the frame-by-frame constraint, our video generator surpasses state-of-the-art baselines on VBench while achieving faster inference and enabling seamless scaling to minute-long sequences. We hope this work will motivate rethinking sequence decomposition in video and other spatiotemporal domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24081 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Autoregressive Video Generation beyond Next Frames Prediction Ren, Sucheng Chen, Chen Wang, Zhenbang Song, Liangchen Zhu, Xiangxin Yuille, Alan Yang, Yinfei Lu, Jiasen Computer Vision and Pattern Recognition Autoregressive models for video generation typically operate frame-by-frame, extending next-token prediction from language to video's temporal dimension. We question that unlike word as token is universally agreed in language if frame is a appropriate prediction unit? To address this, we present VideoAR, a unified framework that supports a spectrum of prediction units including full frames, key-detail frames, multiscale refinements, and spatiotemporal cubes. Among these designs, we find model video generation using \textit{spatiotemporal} cubes as prediction units, which allows autoregressive models to operate across both spatial and temporal dimensions simultaneously. This approach eliminates the assumption that frames are the natural atomic units for video autoregression. We evaluate VideoAR across diverse prediction strategies, finding that cube-based prediction consistently delivers superior quality, speed, and temporal coherence. By removing the frame-by-frame constraint, our video generator surpasses state-of-the-art baselines on VBench while achieving faster inference and enabling seamless scaling to minute-long sequences. We hope this work will motivate rethinking sequence decomposition in video and other spatiotemporal domains. |
| title | Autoregressive Video Generation beyond Next Frames Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.24081 |