EVA: An Embodied World Model for Future Video Anticipation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910997057372160 |
|---|---|
| author | Chi, Xiaowei Fan, Chun-Kai Zhang, Hengyuan Qi, Xingqun Zhang, Rongyu Chen, Anthony Chan, Chi-min Xue, Wei Liu, Qifeng Zhang, Shanghang Guo, Yike |
| author_facet | Chi, Xiaowei Fan, Chun-Kai Zhang, Hengyuan Qi, Xingqun Zhang, Rongyu Chen, Anthony Chan, Chi-min Xue, Wei Liu, Qifeng Zhang, Shanghang Guo, Yike |
| contents | Video generation models have made significant progress in simulating future states, showcasing their potential as world simulators in embodied scenarios. However, existing models often lack robust understanding, limiting their ability to perform multi-step predictions or handle Out-of-Distribution (OOD) scenarios. To address this challenge, we propose the Reflection of Generation (RoG), a set of intermediate reasoning strategies designed to enhance video prediction. It leverages the complementary strengths of pre-trained vision-language and video generation models, enabling them to function as a world model in embodied scenarios. To support RoG, we introduce Embodied Video Anticipation Benchmark(EVA-Bench), a comprehensive benchmark that evaluates embodied world models across diverse tasks and scenarios, utilizing both in-domain and OOD datasets. Building on this foundation, we devise a world model, Embodied Video Anticipator (EVA), that follows a multistage training paradigm to generate high-fidelity video frames and apply an autoregressive strategy to enable adaptive generalization for longer video sequences. Extensive experiments demonstrate the efficacy of EVA in various downstream tasks like video generation and robotics, thereby paving the way for large-scale pre-trained models in real-world video prediction applications. The video demos are available at \hyperlink{https://sites.google.com/view/icml-eva}{https://sites.google.com/view/icml-eva}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15461 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EVA: An Embodied World Model for Future Video Anticipation Chi, Xiaowei Fan, Chun-Kai Zhang, Hengyuan Qi, Xingqun Zhang, Rongyu Chen, Anthony Chan, Chi-min Xue, Wei Liu, Qifeng Zhang, Shanghang Guo, Yike Computer Vision and Pattern Recognition Multimedia Robotics Video generation models have made significant progress in simulating future states, showcasing their potential as world simulators in embodied scenarios. However, existing models often lack robust understanding, limiting their ability to perform multi-step predictions or handle Out-of-Distribution (OOD) scenarios. To address this challenge, we propose the Reflection of Generation (RoG), a set of intermediate reasoning strategies designed to enhance video prediction. It leverages the complementary strengths of pre-trained vision-language and video generation models, enabling them to function as a world model in embodied scenarios. To support RoG, we introduce Embodied Video Anticipation Benchmark(EVA-Bench), a comprehensive benchmark that evaluates embodied world models across diverse tasks and scenarios, utilizing both in-domain and OOD datasets. Building on this foundation, we devise a world model, Embodied Video Anticipator (EVA), that follows a multistage training paradigm to generate high-fidelity video frames and apply an autoregressive strategy to enable adaptive generalization for longer video sequences. Extensive experiments demonstrate the efficacy of EVA in various downstream tasks like video generation and robotics, thereby paving the way for large-scale pre-trained models in real-world video prediction applications. The video demos are available at \hyperlink{https://sites.google.com/view/icml-eva}{https://sites.google.com/view/icml-eva}. |
| title | EVA: An Embodied World Model for Future Video Anticipation |
| topic | Computer Vision and Pattern Recognition Multimedia Robotics |
| url | https://arxiv.org/abs/2410.15461 |