Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918533972099072 |
|---|---|
| author | Liu, Jiuming Ni, Chaojun Liu, Mengmeng Peng, Chensheng Wang, Fangjinhua Shen, Sitian Pollefeys, Marc Tomizuka, Masayoshi Tewari, Ayush Kristensson, Per Ola |
| author_facet | Liu, Jiuming Ni, Chaojun Liu, Mengmeng Peng, Chensheng Wang, Fangjinhua Shen, Sitian Pollefeys, Marc Tomizuka, Masayoshi Tewari, Ayush Kristensson, Per Ola |
| contents | With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI, autonomous driving, etc. Through explicitly incorporating user actions into world state transition, recent literature empowers world modeling with interactivity in an action-conditioned video or 3D generation paradigm, further enhancing controllability over world evolutions and facilitating users to freely traverse, manipulate, navigate, and personalize the state evolution. In this paper, we aim to systematically review recent research trends, technical developments, evaluation benchmarks, and also propose future potential directions in interactive world modeling. Specifically, we first summarize recent efforts and trends in terms of application scenarios, world state evolution, and scene modality. Afterwards, we delve into three crucial technical challenges, including action-conditioned controllability, long-horizon interactions and memory, and action-following responsiveness for real-time interactivity. Furthermore, we also thoroughly compare existing benchmarks and metrics in four specific application fields: open-world exploration, game engine, autonomous driving, and robotics. Finally, we discuss several promising future directions in achieving next-generation interactive world modeling. The corresponding repository is publicly available at: https://github.com/liujiuming123/Awesome-Interactive-World-Model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01164 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends Liu, Jiuming Ni, Chaojun Liu, Mengmeng Peng, Chensheng Wang, Fangjinhua Shen, Sitian Pollefeys, Marc Tomizuka, Masayoshi Tewari, Ayush Kristensson, Per Ola Computer Vision and Pattern Recognition With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI, autonomous driving, etc. Through explicitly incorporating user actions into world state transition, recent literature empowers world modeling with interactivity in an action-conditioned video or 3D generation paradigm, further enhancing controllability over world evolutions and facilitating users to freely traverse, manipulate, navigate, and personalize the state evolution. In this paper, we aim to systematically review recent research trends, technical developments, evaluation benchmarks, and also propose future potential directions in interactive world modeling. Specifically, we first summarize recent efforts and trends in terms of application scenarios, world state evolution, and scene modality. Afterwards, we delve into three crucial technical challenges, including action-conditioned controllability, long-horizon interactions and memory, and action-following responsiveness for real-time interactivity. Furthermore, we also thoroughly compare existing benchmarks and metrics in four specific application fields: open-world exploration, game engine, autonomous driving, and robotics. Finally, we discuss several promising future directions in achieving next-generation interactive world modeling. The corresponding repository is publicly available at: https://github.com/liujiuming123/Awesome-Interactive-World-Model. |
| title | Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2606.01164 |