PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911832342528000 |
|---|---|
| author | Zheng, Yang Zhao, Qingqing Yang, Guandao Yifan, Wang Xiang, Donglai Dubost, Florian Lagun, Dmitry Beeler, Thabo Tombari, Federico Guibas, Leonidas Wetzstein, Gordon |
| author_facet | Zheng, Yang Zhao, Qingqing Yang, Guandao Yifan, Wang Xiang, Donglai Dubost, Florian Lagun, Dmitry Beeler, Thabo Tombari, Federico Guibas, Leonidas Wetzstein, Gordon |
| contents | Modeling and rendering photorealistic avatars is of crucial importance in many applications. Existing methods that build a 3D avatar from visual observations, however, struggle to reconstruct clothed humans. We introduce PhysAvatar, a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data along with the physical parameters of the fabric of their clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for spatio-temporal mesh tracking as well as a physically based inverse renderer to estimate the intrinsic material properties. PhysAvatar integrates a physics simulator to estimate the physical parameters of the garments using gradient-based optimization in a principled manner. These novel capabilities enable PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under motions and lighting conditions not seen in the training data. This marks a significant advancement towards modeling photorealistic digital humans using physically based inverse rendering with physics in the loop. Our project website is at: https://qingqing-zhao.github.io/PhysAvatar |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04421 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations Zheng, Yang Zhao, Qingqing Yang, Guandao Yifan, Wang Xiang, Donglai Dubost, Florian Lagun, Dmitry Beeler, Thabo Tombari, Federico Guibas, Leonidas Wetzstein, Gordon Graphics Computer Vision and Pattern Recognition Modeling and rendering photorealistic avatars is of crucial importance in many applications. Existing methods that build a 3D avatar from visual observations, however, struggle to reconstruct clothed humans. We introduce PhysAvatar, a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data along with the physical parameters of the fabric of their clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for spatio-temporal mesh tracking as well as a physically based inverse renderer to estimate the intrinsic material properties. PhysAvatar integrates a physics simulator to estimate the physical parameters of the garments using gradient-based optimization in a principled manner. These novel capabilities enable PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under motions and lighting conditions not seen in the training data. This marks a significant advancement towards modeling photorealistic digital humans using physically based inverse rendering with physics in the loop. Our project website is at: https://qingqing-zhao.github.io/PhysAvatar |
| title | PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations |
| topic | Graphics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.04421 |