Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.23368 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916673839169536 |
|---|---|
| author | Yang, Xindi Li, Baolu Zhang, Yiming Yin, Zhenfei Bai, Lei Ma, Liqian Wang, Zhiyong Cai, Jianfei Wong, Tien-Tsin Lu, Huchuan Jia, Xu |
| author_facet | Yang, Xindi Li, Baolu Zhang, Yiming Yin, Zhenfei Bai, Lei Ma, Liqian Wang, Zhiyong Cai, Jianfei Wong, Tien-Tsin Lu, Huchuan Jia, Xu |
| contents | Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos and drawing the attention of the community in their potential as world simulators. However, despite their capabilities, VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics, resulting in incorrect dynamics and event sequences. To address this limitation, we propose a novel two-stage image-to-video generation framework that explicitly incorporates physics with vision and language informed physical prior. In the first stage, we employ a Vision Language Model (VLM) as a coarse-grained motion planner, integrating chain-of-thought and physics-aware reasoning to predict a rough motion trajectories/changes that approximate real-world physical dynamics while ensuring the inter-frame consistency. In the second stage, we use the predicted motion trajectories/changes to guide the video generation of a VDM. As the predicted motion trajectories/changes are rough, noise is added during inference to provide freedom to the VDM in generating motion with more fine details. Extensive experimental results demonstrate that our framework can produce physically plausible motion, and comparative evaluations highlight the notable superiority of our approach over existing methods. More video results are available on our Project Page: https://madaoer.github.io/projects/physically_plausible_video_generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23368 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior Yang, Xindi Li, Baolu Zhang, Yiming Yin, Zhenfei Bai, Lei Ma, Liqian Wang, Zhiyong Cai, Jianfei Wong, Tien-Tsin Lu, Huchuan Jia, Xu Computer Vision and Pattern Recognition Artificial Intelligence Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos and drawing the attention of the community in their potential as world simulators. However, despite their capabilities, VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics, resulting in incorrect dynamics and event sequences. To address this limitation, we propose a novel two-stage image-to-video generation framework that explicitly incorporates physics with vision and language informed physical prior. In the first stage, we employ a Vision Language Model (VLM) as a coarse-grained motion planner, integrating chain-of-thought and physics-aware reasoning to predict a rough motion trajectories/changes that approximate real-world physical dynamics while ensuring the inter-frame consistency. In the second stage, we use the predicted motion trajectories/changes to guide the video generation of a VDM. As the predicted motion trajectories/changes are rough, noise is added during inference to provide freedom to the VDM in generating motion with more fine details. Extensive experimental results demonstrate that our framework can produce physically plausible motion, and comparative evaluations highlight the notable superiority of our approach over existing methods. More video results are available on our Project Page: https://madaoer.github.io/projects/physically_plausible_video_generation. |
| title | VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2503.23368 |