Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.12940 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912713331965952 |
|---|---|
| author | Chen, Taiye Ding, Zihan Li, Anjian Zhang, Christina Xiao, Zeqi Wang, Yisen Jin, Chi |
| author_facet | Chen, Taiye Ding, Zihan Li, Anjian Zhang, Christina Xiao, Zeqi Wang, Yisen Jin, Chi |
| contents | Recent advancements in video generation have demonstrated the potential of using video diffusion models as world models, with autoregressive generation of infinitely long videos through masked conditioning. However, such models, usually with local full attention, lack effective memory compression and retrieval for long-term generation beyond the window size, leading to issues of forgetting and spatiotemporal inconsistencies. To enhance the retention of historical information within a fixed memory budget, we introduce a recurrent neural network (RNN) into the diffusion transformer framework. Specifically, a diffusion model incorporating LSTM with attention achieves comparable performance to state-of-the-art RNN blocks, such as TTT and Mamba2. Moreover, existing diffusion-RNN approaches often suffer from performance degradation due to training-inference gap or the lack of overlap across windows. To address these limitations, we propose a novel Recurrent Autoregressive Diffusion (RAD) framework, which executes frame-wise autoregression for memory update and retrieval, consistently across training and inference time. Experiments on Memory Maze and Minecraft datasets demonstrate the superiority of RAD for long video generation, highlighting the efficiency of LSTM in sequence modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12940 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention Chen, Taiye Ding, Zihan Li, Anjian Zhang, Christina Xiao, Zeqi Wang, Yisen Jin, Chi Computer Vision and Pattern Recognition Recent advancements in video generation have demonstrated the potential of using video diffusion models as world models, with autoregressive generation of infinitely long videos through masked conditioning. However, such models, usually with local full attention, lack effective memory compression and retrieval for long-term generation beyond the window size, leading to issues of forgetting and spatiotemporal inconsistencies. To enhance the retention of historical information within a fixed memory budget, we introduce a recurrent neural network (RNN) into the diffusion transformer framework. Specifically, a diffusion model incorporating LSTM with attention achieves comparable performance to state-of-the-art RNN blocks, such as TTT and Mamba2. Moreover, existing diffusion-RNN approaches often suffer from performance degradation due to training-inference gap or the lack of overlap across windows. To address these limitations, we propose a novel Recurrent Autoregressive Diffusion (RAD) framework, which executes frame-wise autoregression for memory update and retrieval, consistently across training and inference time. Experiments on Memory Maze and Minecraft datasets demonstrate the superiority of RAD for long video generation, highlighting the efficiency of LSTM in sequence modeling. |
| title | Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.12940 |