DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908166550192128 |
|---|---|
| author | Huang, Tianyu Zhang, Haoze Zeng, Yihan Zhang, Zhilu Li, Hui Zuo, Wangmeng Lau, Rynson W. H. |
| author_facet | Huang, Tianyu Zhang, Haoze Zeng, Yihan Zhang, Zhilu Li, Hui Zuo, Wangmeng Lau, Rynson W. H. |
| contents | Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation of video generative models, which, however, tends to produce 3D videos with small and discontinuous motions due to the inappropriate extraction and application of physics priors. In this work, to combine the strengths and complementing shortcomings of the above two solutions, we propose to learn the physical properties of a material field with video diffusion priors, and then utilize a physics-based Material-Point-Method (MPM) simulator to generate 4D content with realistic motions. In particular, we propose motion distillation sampling to emphasize video motion information during distillation. In addition, to facilitate the optimization, we further propose a KAN-based material field with frame boosting. Experimental results demonstrate that our method enjoys more realistic motions than state-of-the-arts do. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01476 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors Huang, Tianyu Zhang, Haoze Zeng, Yihan Zhang, Zhilu Li, Hui Zuo, Wangmeng Lau, Rynson W. H. Computer Vision and Pattern Recognition Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation of video generative models, which, however, tends to produce 3D videos with small and discontinuous motions due to the inappropriate extraction and application of physics priors. In this work, to combine the strengths and complementing shortcomings of the above two solutions, we propose to learn the physical properties of a material field with video diffusion priors, and then utilize a physics-based Material-Point-Method (MPM) simulator to generate 4D content with realistic motions. In particular, we propose motion distillation sampling to emphasize video motion information during distillation. In addition, to facilitate the optimization, we further propose a KAN-based material field with frame boosting. Experimental results demonstrate that our method enjoys more realistic motions than state-of-the-arts do. |
| title | DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.01476 |