GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910271310659584 |
|---|---|
| author | Zhou, Kaichen Chen, Yuzhen Zhan, Fangneng Hua, Hang Chen, Grace Chang, Xinhai Qu, Ao Du, Yilun Liu, Zhuang Liang, Paul Pu Wang, Mengyu |
| author_facet | Zhou, Kaichen Chen, Yuzhen Zhan, Fangneng Hua, Hang Chen, Grace Chang, Xinhai Qu, Ao Du, Yilun Liu, Zhuang Liang, Paul Pu Wang, Mengyu |
| contents | Video world models can generate realistic futures from a single instruction, but they often fail to track the same physical points consistently across time. As a result, the generated videos appear plausible, yet lack the physical grounding required for reliable action execution, such as robot manipulation. We present GEM-4D, a geometry-grounded video world model that resolves this limitation by injecting dense 4D correspondence supervision distilled from a pretrained geometry foundation model into the video generative backbone during training. This supervision enables the model to jointly capture appearance and geometric structure while retaining a single-stream architecture with no additional inference cost. We further introduce an inverse dynamics module that converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-art performance on both video prediction and geometric consistency across both simulation and realistic scenarios and improves real-world manipulation success from 61% to 81%. Additional results are available at https://gem-4d.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_22882 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation Zhou, Kaichen Chen, Yuzhen Zhan, Fangneng Hua, Hang Chen, Grace Chang, Xinhai Qu, Ao Du, Yilun Liu, Zhuang Liang, Paul Pu Wang, Mengyu Computer Vision and Pattern Recognition Robotics Video world models can generate realistic futures from a single instruction, but they often fail to track the same physical points consistently across time. As a result, the generated videos appear plausible, yet lack the physical grounding required for reliable action execution, such as robot manipulation. We present GEM-4D, a geometry-grounded video world model that resolves this limitation by injecting dense 4D correspondence supervision distilled from a pretrained geometry foundation model into the video generative backbone during training. This supervision enables the model to jointly capture appearance and geometric structure while retaining a single-stream architecture with no additional inference cost. We further introduce an inverse dynamics module that converts correspondence-consistent video rollouts into executable robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-art performance on both video prediction and geometric consistency across both simulation and realistic scenarios and improves real-world manipulation success from 61% to 81%. Additional results are available at https://gem-4d.github.io/. |
| title | GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2605.22882 |