I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916185303416832 |
|---|---|
| author | Zhao, Chengfeng Zhang, Juze Du, Jiashen Shan, Ziwei Wang, Junye Yu, Jingyi Wang, Jingya Xu, Lan |
| author_facet | Zhao, Chengfeng Zhang, Juze Du, Jiashen Shan, Ziwei Wang, Junye Yu, Jingyi Wang, Jingya Xu, Lan |
| contents | We are living in a world surrounded by diverse and "smart" devices with rich modalities of sensing ability. Conveniently capturing the interactions between us humans and these objects remains far-reaching. In this paper, we present I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting: using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU). It combines general motion inference and category-aware refinement. For the former, we introduce a holistic human-object tracking method to fuse the IMU signals and the RGB stream and progressively recover the human motions and subsequently the companion object motions. For the latter, we tailor a category-aware motion diffusion model, which is conditioned on both the raw IMU observations and the results from the previous stage under over-parameterization representation. It significantly refines the initial results and generates vivid body, hand, and object motions. Moreover, we contribute a large dataset with ground truth human and object motions, dense RGB inputs, and rich object-mounted IMU measurements. Extensive experiments demonstrate the effectiveness of I'm-HOI under a hybrid capture setting. Our dataset and code will be released to the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_08869 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions Zhao, Chengfeng Zhang, Juze Du, Jiashen Shan, Ziwei Wang, Junye Yu, Jingyi Wang, Jingya Xu, Lan Computer Vision and Pattern Recognition We are living in a world surrounded by diverse and "smart" devices with rich modalities of sensing ability. Conveniently capturing the interactions between us humans and these objects remains far-reaching. In this paper, we present I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting: using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU). It combines general motion inference and category-aware refinement. For the former, we introduce a holistic human-object tracking method to fuse the IMU signals and the RGB stream and progressively recover the human motions and subsequently the companion object motions. For the latter, we tailor a category-aware motion diffusion model, which is conditioned on both the raw IMU observations and the results from the previous stage under over-parameterization representation. It significantly refines the initial results and generates vivid body, hand, and object motions. Moreover, we contribute a large dataset with ground truth human and object motions, dense RGB inputs, and rich object-mounted IMU measurements. Extensive experiments demonstrate the effectiveness of I'm-HOI under a hybrid capture setting. Our dataset and code will be released to the community. |
| title | I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.08869 |