Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915714178220032 |
|---|---|
| author | Fan, Chun-Kai Chi, Xiaowei Ju, Xiaozhu Li, Hao Bao, Yong Wang, Yu-Kai Chen, Lizhang Jiang, Zhiyuan Ge, Kuangzhi Li, Ying Mi, Weishi Wuwu, Qingpo Jia, Peidong Luo, Yulin Zhang, Kevin Qin, Zhiyuan Dai, Yong Han, Sirui Guo, Yike Zhang, Shanghang Tang, Jian |
| author_facet | Fan, Chun-Kai Chi, Xiaowei Ju, Xiaozhu Li, Hao Bao, Yong Wang, Yu-Kai Chen, Lizhang Jiang, Zhiyuan Ge, Kuangzhi Li, Ying Mi, Weishi Wuwu, Qingpo Jia, Peidong Luo, Yulin Zhang, Kevin Qin, Zhiyuan Dai, Yong Han, Sirui Guo, Yike Zhang, Shanghang Tang, Jian |
| contents | As world models gain momentum in Embodied AI, an increasing number of works explore using video foundation models as predictive world models for downstream embodied tasks like 3D prediction or interactive generation. However, before exploring these downstream tasks, video foundation models still have two critical questions unanswered: (1) whether their generative generalization is sufficient to maintain perceptual fidelity in the eyes of human observers, and (2) whether they are robust enough to serve as a universal prior for real-world embodied agents. To provide a standardized framework for answering these questions, we introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val). Building upon 609 robot manipulation data, Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization, and execution. We propose a comprehensive evaluation protocol with 22 metrics to assess the models' generation ability, which achieves a high Pearson Correlation between the overall score and human preference (>0.93) and establishes a reliable foundation for the Human Turing Test. On Wow-wo-val, models achieve only 17.27 on long-horizon planning and at best 68.02 on physical consistency, indicating limited spatiotemporal consistency and physical reasoning. For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world. However, most models collapse to $\approx$ 0% success, while WoW maintains a 40.74% success rate. These findings point to a noticeable gap between the generated videos and the real world, highlighting the urgency and necessity of benchmarking World Model in Embodied AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04137 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test Fan, Chun-Kai Chi, Xiaowei Ju, Xiaozhu Li, Hao Bao, Yong Wang, Yu-Kai Chen, Lizhang Jiang, Zhiyuan Ge, Kuangzhi Li, Ying Mi, Weishi Wuwu, Qingpo Jia, Peidong Luo, Yulin Zhang, Kevin Qin, Zhiyuan Dai, Yong Han, Sirui Guo, Yike Zhang, Shanghang Tang, Jian Robotics Artificial Intelligence Computer Vision and Pattern Recognition As world models gain momentum in Embodied AI, an increasing number of works explore using video foundation models as predictive world models for downstream embodied tasks like 3D prediction or interactive generation. However, before exploring these downstream tasks, video foundation models still have two critical questions unanswered: (1) whether their generative generalization is sufficient to maintain perceptual fidelity in the eyes of human observers, and (2) whether they are robust enough to serve as a universal prior for real-world embodied agents. To provide a standardized framework for answering these questions, we introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val). Building upon 609 robot manipulation data, Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization, and execution. We propose a comprehensive evaluation protocol with 22 metrics to assess the models' generation ability, which achieves a high Pearson Correlation between the overall score and human preference (>0.93) and establishes a reliable foundation for the Human Turing Test. On Wow-wo-val, models achieve only 17.27 on long-horizon planning and at best 68.02 on physical consistency, indicating limited spatiotemporal consistency and physical reasoning. For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world. However, most models collapse to $\approx$ 0% success, while WoW maintains a 40.74% success rate. These findings point to a noticeable gap between the generated videos and the real world, highlighting the urgency and necessity of benchmarking World Model in Embodied AI. |
| title | Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.04137 |