Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.22626 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914221998997504 |
|---|---|
| author | Gu, Yuming Wang, Yizhi Hong, Yining Gao, Yipeng Jiang, Hao Wang, Angtian Liu, Bo Dennler, Nathaniel S. Kuang, Zhengfei Li, Hao Wetzstein, Gordon Ma, Chongyang |
| author_facet | Gu, Yuming Wang, Yizhi Hong, Yining Gao, Yipeng Jiang, Hao Wang, Angtian Liu, Bo Dennler, Nathaniel S. Kuang, Zhengfei Li, Hao Wetzstein, Gordon Ma, Chongyang |
| contents | Embodied visual planning aims to enable manipulation tasks by imagining how a scene evolves toward a desired goal and using the imagined trajectories to guide actions. Video diffusion models, through their image-to-video generation capability, provide a promising foundation for such visual imagination. However, existing approaches are largely forward predictive, generating trajectories conditioned on the initial observation without explicit goal modeling, thus often leading to spatial drift and goal misalignment. To address these challenges, we propose Envision, a diffusion-based framework that performs visual planning for embodied agents. By explicitly constraining the generation with a goal image, our method enforces physical plausibility and goal consistency throughout the generated trajectory. Specifically, Envision operates in two stages. First, a Goal Imagery Model identifies task-relevant regions, performs region-aware cross attention between the scene and the instruction, and synthesizes a coherent goal image that captures the desired outcome. Then, an Env-Goal Video Model, built upon a first-and-last-frame-conditioned video diffusion model (FL2V), interpolates between the initial observation and the goal image, producing smooth and physically plausible video trajectories that connect the start and goal states. Experiments on object manipulation and image editing benchmarks demonstrate that Envision achieves superior goal alignment, spatial consistency, and object preservation compared to baselines. The resulting visual plans can directly support downstream robotic planning and control, providing reliable guidance for embodied agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22626 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion Gu, Yuming Wang, Yizhi Hong, Yining Gao, Yipeng Jiang, Hao Wang, Angtian Liu, Bo Dennler, Nathaniel S. Kuang, Zhengfei Li, Hao Wetzstein, Gordon Ma, Chongyang Computer Vision and Pattern Recognition Embodied visual planning aims to enable manipulation tasks by imagining how a scene evolves toward a desired goal and using the imagined trajectories to guide actions. Video diffusion models, through their image-to-video generation capability, provide a promising foundation for such visual imagination. However, existing approaches are largely forward predictive, generating trajectories conditioned on the initial observation without explicit goal modeling, thus often leading to spatial drift and goal misalignment. To address these challenges, we propose Envision, a diffusion-based framework that performs visual planning for embodied agents. By explicitly constraining the generation with a goal image, our method enforces physical plausibility and goal consistency throughout the generated trajectory. Specifically, Envision operates in two stages. First, a Goal Imagery Model identifies task-relevant regions, performs region-aware cross attention between the scene and the instruction, and synthesizes a coherent goal image that captures the desired outcome. Then, an Env-Goal Video Model, built upon a first-and-last-frame-conditioned video diffusion model (FL2V), interpolates between the initial observation and the goal image, producing smooth and physically plausible video trajectories that connect the start and goal states. Experiments on object manipulation and image editing benchmarks demonstrate that Envision achieves superior goal alignment, spatial consistency, and object preservation compared to baselines. The resulting visual plans can directly support downstream robotic planning and control, providing reliable guidance for embodied agents. |
| title | Envision: Embodied Visual Planning via Goal-Imagery Video Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.22626 |