Video World Models with Long-term Spatial Memory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910989848412160 |
|---|---|
| author | Wu, Tong Yang, Shuai Po, Ryan Xu, Yinghao Liu, Ziwei Lin, Dahua Wetzstein, Gordon |
| author_facet | Wu, Tong Yang, Shuai Po, Ryan Xu, Yinghao Liu, Ziwei Lin, Dahua Wetzstein, Gordon |
| contents | Emerging world models autoregressively generate video frames in response to actions, such as camera movements and text prompts, among other control signals. Due to limited temporal context window sizes, these models often struggle to maintain scene consistency during revisits, leading to severe forgetting of previously generated environments. Inspired by the mechanisms of human memory, we introduce a novel framework to enhancing long-term consistency of video world models through a geometry-grounded long-term spatial memory. Our framework includes mechanisms to store and retrieve information from the long-term spatial memory and we curate custom datasets to train and evaluate world models with explicitly stored 3D memory mechanisms. Our evaluations show improved quality, consistency, and context length compared to relevant baselines, paving the way towards long-term consistent world generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_05284 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Video World Models with Long-term Spatial Memory Wu, Tong Yang, Shuai Po, Ryan Xu, Yinghao Liu, Ziwei Lin, Dahua Wetzstein, Gordon Computer Vision and Pattern Recognition Emerging world models autoregressively generate video frames in response to actions, such as camera movements and text prompts, among other control signals. Due to limited temporal context window sizes, these models often struggle to maintain scene consistency during revisits, leading to severe forgetting of previously generated environments. Inspired by the mechanisms of human memory, we introduce a novel framework to enhancing long-term consistency of video world models through a geometry-grounded long-term spatial memory. Our framework includes mechanisms to store and retrieve information from the long-term spatial memory and we curate custom datasets to train and evaluate world models with explicitly stored 3D memory mechanisms. Our evaluations show improved quality, consistency, and context length compared to relevant baselines, paving the way towards long-term consistent world generation. |
| title | Video World Models with Long-term Spatial Memory |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.05284 |