WHAC: World-grounded Humans and Cameras
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911803678654464 |
|---|---|
| author | Yin, Wanqi Cai, Zhongang Wang, Ruisi Wang, Fanzhou Wei, Chen Mei, Haiyi Xiao, Weiye Yang, Zhitao Sun, Qingping Yamashita, Atsushi Liu, Ziwei Yang, Lei |
| author_facet | Yin, Wanqi Cai, Zhongang Wang, Ruisi Wang, Fanzhou Wei, Chen Mei, Haiyi Xiao, Weiye Yang, Zhitao Sun, Qingping Yamashita, Atsushi Liu, Ziwei Yang, Lei |
| contents | Estimating human and camera trajectories with accurate scale in the world coordinate system from a monocular video is a highly desirable yet challenging and ill-posed problem. In this study, we aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly, by leveraging the synergy between three critical players: the world, the human, and the camera. Our approach is founded on two key observations. Firstly, camera-frame SMPL-X estimation methods readily recover absolute human depth. Secondly, human motions inherently provide absolute spatial cues. By integrating these insights, we introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation (EHPS) alongside camera pose estimation, without relying on traditional optimization techniques. Additionally, we present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras, and features diverse interactive human motions as well as realistic camera trajectories. Extensive experiments on both standard and newly established benchmarks highlight the superiority and efficacy of our framework. We will make the code and dataset publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12959 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WHAC: World-grounded Humans and Cameras Yin, Wanqi Cai, Zhongang Wang, Ruisi Wang, Fanzhou Wei, Chen Mei, Haiyi Xiao, Weiye Yang, Zhitao Sun, Qingping Yamashita, Atsushi Liu, Ziwei Yang, Lei Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning Robotics Estimating human and camera trajectories with accurate scale in the world coordinate system from a monocular video is a highly desirable yet challenging and ill-posed problem. In this study, we aim to recover expressive parametric human models (i.e., SMPL-X) and corresponding camera poses jointly, by leveraging the synergy between three critical players: the world, the human, and the camera. Our approach is founded on two key observations. Firstly, camera-frame SMPL-X estimation methods readily recover absolute human depth. Secondly, human motions inherently provide absolute spatial cues. By integrating these insights, we introduce a novel framework, referred to as WHAC, to facilitate world-grounded expressive human pose and shape estimation (EHPS) alongside camera pose estimation, without relying on traditional optimization techniques. Additionally, we present a new synthetic dataset, WHAC-A-Mole, which includes accurately annotated humans and cameras, and features diverse interactive human motions as well as realistic camera trajectories. Extensive experiments on both standard and newly established benchmarks highlight the superiority and efficacy of our framework. We will make the code and dataset publicly available. |
| title | WHAC: World-grounded Humans and Cameras |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Graphics Machine Learning Robotics |
| url | https://arxiv.org/abs/2403.12959 |